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Preface 



The fourth international workshop, “Engineering Societies in the Agents World” 
(ESAW 2003) was a tlrree-day event that took place at the end of October 2003. 
After previous events in Germany, the Czech Republic, and Spain, the workshop 
crossed the Channel, to be held at the premises of Imperial College, London. 

The steady increase in the variety of backgrounds of contributing scien- 
tists, fascinating new perspectives on the topics, and number of participants, 
bespeaks the success of the ESAW workshop series. Its idea was born in 1999 
among members of the working group on “Communication, Coordination, and 
Collaboration” of the first lease of life of the European Network of Excellence 
on Agent-Based Computing, AgentLink, out of a critical discussion about the 
general mindset of the agent community. At that time, we felt that proper con- 
siderations of systemic aspects of agent technology deployment, such as acknowl- 
edgement of the importance of the social and environmental perspectives, were 
sorely missing: a deficiency that we resolved should be addressed directly by a 
new forum. 

A first focal point was the vision that to tackle the issues inherently con- 
nected to the emergent complexity of multi-agent systems (MAS) it would be 
inevitable to introduce the notion of a society of agents as a first-class entity in 
the modeling and engineering of MAS. In particular, paying attention to soft- 
ware infrastructure as a location to provide intelligence in MAS, and the notion 
of social intelligence drove the first ESAW workshop, co-located with ECAI 2000 
in Berlin. ESAW 2001, held in Prague together with the by now renowned Euro- 
pean Agent Systems Summer School (ACAI’01), reinforced the line of research 
relating to the design of agent society and underlined further the necessity for 
methodologies to properly guide the increasingly popular use of social and cogni- 
tive concepts in agent theories and technologies. The third workshop in Madrid 
took advantage of co-location with the workshop series on Cooperative Informa- 
tion Agents (CIA 2002) to set itself apart and gain further in identity by opening 
up to a yet a wider range of contributing technologies, while maintaining its cen- 
tral focus on theoretical and methodological aspects applied by this direction of 
research. 

ESAW 2003 was the first workshop neither connected to nor co-located with 
any other scientific event. Even so, the stand-alone workshop proved the most 
vivid and rich one to date: this stands to testify, on the one hand, that the 
community aggregating around ESAW is by now sufficiently large and mature to 
sustain an autonomous scientific event, and, on the other hand, that the original 
intent to define and extend this community beyond the traditional (and already 
outdated) borders of computer engineering has definitely been met. Following 
the tradition of this workshop, the structure of the event developed around a 
few main themes constituting the pivotal elements of the sessions, as well as 
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two invited talks that provided further topics of high relevance and prompted 
stimulating discussions. 

In particular, the sessions addressed the following themes over the three days 
of the workshop: 

— Agent- Oriented Software Engineering and Formal Methods. Two sessions 
covered methodological aspects of AOSE as well as formal methods in the 
analysis and planning of agent systems. 

— MAS Protocols and Interaction Management. This session hosted presenta- 
tions and discussions on communication and coordination between agents, 
focusing in particular on social aspects. 

— MAS Organization and Workflow. Presentations in this session concentrated 
on organizational aspects, as well as related technologies and applications. 

— MAS Architectures, Cooperation and Teamwork. In this session we discussed 
agent architectures, team and coalition formation, and economies of interac- 
tion in agent societies. 

— Artificial Intelligence Techniques in MAS. This session covered more tra- 
ditional (in a certain sense) topics in MAS research, such as planning and 
collective forms of intelligence. 

— Agent Society Dynamics and Engineering. This session developed a series of 
notions from as heterogeneous backgrounds as sociology, political philosophy, 
and organizational theories, to serve as sources for foundational concepts for 
agent societies and their construction. 

— Agent Applications: Services, User Modeling, and E-Commerce. In this ses- 
sion, different applications of agent technologies were presented, including 
user profiling, intelligent and dynamic service integration, and the realiza- 
tion of models of trust. 

Two invited presentations rounded off the program in a most worthy manner. Dr. 
Andre P. Meyer, of the Command & Control and Simulation group of the Dutch 
TNO FEL, spoke about “Privacy-Aware Mobile Agents: Protecting Privacy by 
Modeling Social Behavior in Open Systems of Software Agents,” examining the 
problem of privacy in open systems where a multitude of different MAS may 
interact. The presentation by Dr. Jean-Pierre Muller, now senior researcher at 
the French LIRMM, entitled “Emergence of Collective Behavior; Simulation and 
Social Engineering,” examined the different notions of emergent behavior in 
complex systems and the correlation with concepts such as agent orientated 
and environment oriented programming. Techniques discussed in the analysis 
phase have been applied to resource management tasks - in particular, social 
engineering and ecological modeling. 

It is useful to underline how the tradition of ESAW differs from other sci- 
entific workshops: here, the selection process includes the very meeting event, 
to which in particular the typical borderline paper submissions are also invited: 
The workshop allows the presenters to utilize the open atmosphere of discussion 
(promoted as much as possible by the organizers) to get their own innovative 
contributions into focus and make them stand out as deserved, whilst, at the 
same time, offering the organizers in their role as curators of the event the 
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possibility to catalyze and encourage original approaches and judge individual 
efforts in a more comprehensive way. The constructive quality of the workshop 
- in particular, of this most recent event - thus contrasts decidedly with the 
dry and meticulous climate that all too often characterises analogous events, 
and it thereby promotes a typically broader and more collaborative scientific 
development of presented work. 

The complete range of contributions that were collected in the working notes 
of the event are available online - along with the presentation slides - at the 
ESAW 2003 workshop site. The present post-proceedings continue the series pub- 
lished with Springer- Verlag (ESAW 2000: LNAI 1972, ESAW 2001: LNAI 2203, 
and ESAW 2002: LNAI 2577). This volume contains reworked and extended 
versions of selected papers and also includes contributions by the two invited 
speakers. 

The organizers gratefully acknowledge financial support granted by the fol- 
lowing institutions: 

— Polo Scientifico-Didattico di Cesena, Universita degli Studi di Bologna 

— Imperial College London 

— the Austrian Society for Artificial Intelligence (OGAI) 

— Whitestein Technologies 

as well as the scientific support by ACM SIGART and AgentLink II. Our thanks 
also go to Springer- Verlag’s Alfred Hofmann for his essential background role in 
helping ESAW through its infancy. The Austrian Research Institute for Artificial 
Intelligence is supported by the Austrian Federal Ministry for Education, Science 
and Culture and by the Austrian Federal Ministry for Transport, Innovation and 
Technology. 

The next ESAW workshop is scheduled to be hosted in France by the Uni- 
versity of Toulouse in October 2004, with Marie-Pierre Gleizes, Andrea Omicini, 
and Franco Zambonelli as organizers. We look forward to an ever broader and 
larger attendance, an even more lively interaction, and a still higher level of 
originality and innovation. 



April 2004 



Andrea Omicini 
Paolo Petta 
Jeremy Pitt 
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Abstract. The goal of this paper is to explore the notion of complex sys- 
tem and, in particular the emergence phenomenon, in order to see which 
lessons could be learned for both understanding and designing complex 
software systems. Complex systems are described as sets of non-linearly 
interacting components making multi-agent systems particularly suit- 
able for modelling and designing such systems. The notion of emergence 
is explicited and used to derive ways of understanding and designing 
such complex systems. We conclude by discussing the pros and cons of 
the emergentist approaches and the research perspectives. 



1 Introduction 

As explicited in the aims and scope of the “Engineering Societies in the Agents 
World” workshop, software systems are undergoing drastic changes in scale and 
complexity, making them more resemble natural systems and societies than me- 
chanical systems and traditional software architectures. The goal of this paper 
is to explore the notion of complex system and, in particular the emergence 
phenomenon, in order to to see which lessons could be learned for both under- 
standing and designing such software systems. 

A traditional approach in computer science is to decompose the system in 
manageable components (functions, objects or coarser grain components) with 
clear interfaces, i.e. manageable way of handling the interactions, most often 
carried out by a middleware (as discussed in [1]). Artificial Intelligence and 
Multi-Agent Systems are proposing problem solving methods inspired by the 
strategies developed by the natural systems (from ant colonies to human beings 
conceived as thinkers). Nevertheless these methods mostly rely on an a priori 
formalisation of the problem domain. In dynamic and uncertain domains, this 
a priori formalisation becomes difficult and requires an increased adaptive ca- 
pability. Nature seems to have solved this problem by emergence of collective 

* I would like to thank the program committee of the ESAW’03 workshop for inviting 
me to this very stimulating event. 

** Also associate researcher to LIRMM, 161, rue Ada, 34392 Montpellier cedex 5 - 
France. 
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behaviours self-organising from the dynamics of entities in interaction between 
themselves and with an environment. Unfortunately, the notion of emergence it- 
self is problematic for the modeller as well as for the designer, being most often 
defined by an absence of “something” as composability, predictability, and so on 
when the notion itself is not criticised. 

The notion of emergence is usually associated with a misunderstanding or 
with an intuition facing some phenomena observed in nature; it is therefore re- 
jected for a reductionist interpretation. For example, Daniel Memmi [2] limits 
emergence to a problem of description or explanation and assumes that emergent 
phenomena are examples among others of the variety of scientific explanations. 
In this sense, emergence is not in nature but in the change of observation focus 
on the phenomena. However, even if emergence can be characterised epistemo- 
logically, these phenomena have to pre-exist to our observation. In other terms, 
the emergence of an entity, a structure, a function or a process is in the system 
independently of our observation even if it is a change of the point of view of 
the observer, which reveals the emergence. 

To better understand and exploit these emergent phenomena, one should pro- 
vide an alternative approach not only to computer modelling, but also to prob- 
lem solving. This approach principle is to build a society of agents, immersed in 
an environment, which by their interactions will evolve towards a stable state 
representing a solution. This approach, initiated by R.Brooks in robotics [3], is 
opposed to the classical problem solving approach where the global resolution 
task is decomposed into subtasks. The program then codes the resolution steps; 
while executing, the process follows the predefined path until the solution is 
reached. In the “emergentist” approach, the program codes the agents, the en- 
vironment and the interactions; while executing, the process self-organises and 
builds a solution. 

Another characteristic of the emergentist approach is the adaptative capa- 
bility of emergent phenomena or structures to the changes of the environment. 
These phenomena are in dynamic interaction with the environment, but are not 
totally dependent on it. Generic regularities and properties are abstracted away 
through self-organisation and are applicable in other environments. In reality, the 
environment instantiates behavioural and structural rules, raising the emergence 
of a global phenomena. For example, a bacteria following a sugar gradient can go 
towards a sugar source or follow such a source without modifying the “program” 
controlling the behaviour of the bacteria. This adaptability to external changes 
is immediate because it does not rely on internal representations nor internal- 
isation of paths. Therefore it does not necessitate updating or modification of 
the representation. 

In the french multi-agent community, the concept of emergence for problem- 
solving has gained considerable interest. We can cite the teams at LIRMM [4, 5], 
LIP 6 [6], LEIBNIZ [7, 8], CASCAD [9] or IRIT [10]. It is less used in the anglo- 
saxon community with the important exception of the work by Van Parunak 
around industrial applications [11]. The notion of emergence is better explored 
in the artificial life domain where we can cite the work of Deneubourg [12], 
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E. Bonabeau [13], J.L. Dessalles, L. Steels [14]. The two papers of [13] : “Char- 
acterizing emergent phenomena” , give a number of examples of phenomena con- 
sidered as emergent and propose a frame of study for a better understanding of 
these phenomena. Finally, we have to make reference to the seminal work of S. 
Forrest [15, 16] in the domain of emergent computation, we shall come to later 
on. 

In the following, we will first introduce the notion of complex system, point- 
ing out the importance of emergence in this definition. Several approaches to 
cope with complex systems shall be reviewed and illustrated. We shall conclude 
that multi-agent systems are good candidates for modelling and designing com- 
plex systems. The next section will introduce the notion of emergence, which is 
of outmost importance to understand complex systems. We shall propose a defi- 
nition, which can be operational for understanding and for designing multi-agent 
systems. In section 4, we will discuss how we can use this definition to derive 
a methodology for designing emergentist multi-agent systems. Section 5 will 
conclude on the interest of the advantages and limits of using an emergentist 
approach to multi-agent systems and open research perspectives. 

2 Complex Systems 

It is hard to find a definition of what a complex system is but a mere list of 
properties (as is the case for intelligence, life, etc.). We can cite the exception of 
the definition of complex adaptive systems in [17] with a bias towards darwinian 
adaptiveness. Roughly speaking (and almost tautologically), a complex system 
is: 

A System: a set of interacting components composing a whole (which whole 
to consider is dependent on the observer, hence on the question being asked) 
creating, de facto , a distinction between the system to be considered and the 
rest (the environment or outside); 

Which is Complex: the interactions among the components are non-linear, 
such that the global behaviour of the system cannot be compositionaly de- 
duced from the components’ behaviours. 

The second property makes the distinction between a complex system and a mere 
complicated system, which may have up to a huge number of components and 
still be compositionaly understood (like the digital electronic circuits, for exam- 
ple). 

This definition exhibits most of the properties generally ascribed from com- 
plex systems [18], i.e.: 

— the need of multi-scale descriptions, because it minimaly implies the articu- 
lation of the level of the components, the level of the whole and the level of 
the underlying environment; 

— the multiplicity of view points because the wholes to consider are intrinsically 
related to the question being asked and therefore give rise to interacting view 
points (in addition to interacting components!); 
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— the intricacy of whole and component behaviours both by the reciprocal rela- 
tionship of the global behaviour and the local behaviours and by the relative 
autonomy of the global behaviour with respect to the local behaviours; 

— the emergence of the whole organisation because of the non-linearity of the 
underlying interactions. 

In order to model complex systems, three approaches can be sketched: 

Analytical Approaches: by analysing the system component by component 
as advocated by the classical rational approach. In such models, the focus is 
on the individual behaviours rather than on the interactions; 

Holistic or Systemic Approaches: by analysing the system as a whole by 
isolating aggregated variables and their interactions (rather than the inter- 
actions between the components) as is done in most dynamical models as 
compartment, statistical or eulerian models; 

Constructivist Approaches: by trying to articulate the individual be- 
haviours of the components with the global behaviour of the system as is 
done in lagrangian dynamical models, individual-based approaches, micro- 
simulation or multi-agent systems depending on the scientific domain in 
which such inquiries take place. In such models, the focus is on the interac- 
tions rather than on the individual behaviours. 

As transdisciplinary considerations, it is not a surprise that these kinds of ap- 
proaches can be found in most scientific fields in a form or another. For example, 
in sociology, we find the distinction between methodological individualism (the 
first approach), methodological holism (the second category) and constructivist 
approaches (see [19]). 

As an example, fluid dynamics can be described from: 

— the behaviour of water molecules: the analytical approach; 

— the Navier-Stockes equations: the holistic approach; 

— the interaction among micro-level entities (as droplets or vortices) and the 
resulting structure of the fluid dynamics: the constructivist approach. 

Each approach has its own domain of validity as negligible non linear interactions 
for the analytical approaches, and sufficiently stable systems for the second ap- 
proach (when the system does not undergo changes, which can result in a change 
of the equations themselves) . The constructivist approach can be used in several 
cases: 

— when one knows the individual behaviour and one wants to explore the 
resulting global dynamics. It is particularly the case for negotiation support 
where the various actors are not always conscious of the joined effect of their 
current and future actions; 

— when one knows or intends (when designing) the global behaviour and one 
seeks an explanation or design from local behaviours. It is the case in science 
where an explanation (rather than a description) must always relate on un- 
derlying phenomena[20] (a stone does not fall because of the gravity law but 
because of the gravitational interaction between two bodies). In this case, 
the constructivist approach has a heuristic role in the scientific inquiry; 



Emergence of Collective Behaviour and Problem Solving 



5 



— when one knows both behaviours but the relationship is hard to articulate as 
it is the case when the whole behaviour emerges from the local interactions 
and when the global behaviour can qualitatively change over time (requiring 
new sets of equations to describe it). In general, the constructivist approach 
is related to any understanding building process. 

As it can be seen, the notion of emergence is at the core of the notion of com- 
plex systems and multi-agent systems at the core of constructivist approaches 
to complex systems. It is the reason why we shall explore the notion of emer- 
gence more deeply, mostly based on the work of a special interest group called 
“COLLINE” 1 [21]. 

3 Notion of Emergence 

As we shall see, the notion of emergence is multiply defined rather than ill- 
defined and covers two aspects : the static aspect (emergence as a result or 
observable) and the dynamic aspect (emergence as a process). Focusing on the 
former aspect, we shall present the history and some of the most interesting 
definitions without being exhaustive. After discussing these definitions, we shall 
propose a new one opening the possibility to derive a methodology for designing 
multi-agent systems, in particular for problem solving. 



The British Emergentism The first mention of the term “emergence” comes 
from the so-called british emergentism. It initialy took place in the debate be- 
tween various accounts of life as: the substantial vitalism , which states that it 
exists a substance linked to the living called entelechy; the mechanistic theory 
for which we are only machines (life is reducible to biochemical processes ); 
and finally the theory of emergentism (Lewes, S. Alexander, Broad, Stuart Mill: 
book of Broad “the mind and its place in nature”, 1923). This last theory is 
based on the work by Stuart Mill, which distinguishes between two types of laws 
organising nature : 

— the homopathic or resultant modality, we can explain by causal laws or com- 
position of them ; 

— the heteropathic or chemical or emergent modality, we cannot explain by 
causal laws as the acquisition of the properties of water out of the properties 
of oxygen and hydrogen respectively. 

The debate is on the existence of these heteropathic laws, which raises the prob- 
lem of relating specific sciences (chemistry, biology, etc.) to physics. 

The argumentation of the emergentists uses a vision of nature in levels artic- 
ulated on top of one another, the physical level being the most fundamental, and 

1 “COLLINE” stands for “COLLective, INteraction, Emergence”, SIG of the Multi- 
agent System chapter of AFIA “Association Franiaise d’Intelligence Artificiellc” (Ar- 
tificial Intelligence French Association). 
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on the ontological existence of the entities at a given level, emerging from the 
level immediately under it. For the british emergentist, the entities at a certain 
level n + 1 emerging from a level n exist if and only if they have a causal power 
and therefore allows the explanation of the phenomena at their level. If we can 
find a cause between n and n + 1, we do not have emergence but reducibility. 
If there is only a cause between n and n + 1, it is an epiphenomenon (e.g. the 
shadow of the hand). In this argumentation, the nature and the existence of 
a descending causality is fundamental. 



Recent Definitions of Emergence 

Emergent Computation. A definition of Stephanie Forrest [16] of the notion of 
emergent computation is particularly interesting because it is conceptually closer 
to what we want to formalise in multi-agent systems. Emergent computation is 
defined as: 

— a set of entities in interaction: the process; 

— an epiphenomenon produced by this process: a stable state, an invariant or 
an execution trace; 

— the interpretation of this epiphenomenon as a computation or the result of 
a computation. 

A first remark concerns the distinction between an epiphenomenon and an emer- 
gence. What is emergent is not the stable state, the invariant or the trace but its 
interpretation in a given vocabulary distinct from the vocabulary in which the 
process is programmed. For example with the ants, the emergent phenomenon is 
not the pheromone trace but its identification by the observer as a path between 
the nest and the food source. 

As a second remark, an epiphenomenon is not an emergence because it as 
been created by the process, but does not interact with the process itself (no 
feedback from the level n + 1 on the level n) . Automatically, we can create an 
emergence precisely when this feedback takes place. In order to do that, it is 
enough that the trace (it is the only physical reality, the stable state or the 
invariant can only be potentially perceivable when they let a trace) interacts 
with the process. At this moment something new is produced, which is neither 
in the process (because we need the trace) nor in the trace (because we need the 
process) and, if it stabilises, it becomes emergent simultaneously as a structure 
(the trace) and as a dynamics (the process). We have an illustration of the 
duality structure/dynamics. 

Emergence in Cognitive Science. John Searle [22] writes : “Some characteristics 
of the system can be deduced or conceived, or computed from the characteristics 
of [its components] on the simple basis of their arrangement or of their compo- 
sition (and sometimes from the relations they have with their environment) 
for example the form, the weight, the speed. But other characteristics cannot 
be conceived only from the composition of its elements or their environmental 
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relations; they must be explained in terms of causal interactions produced be- 
tween the elements. Let us call them the “causally emergent characteristics of 
the system” . Solidity, liquidity and transparency are some examples. From these 
definitions, consciousness is an emergent property of the system. This conception 
of causal emergence, we will call “emergence 1”, must be distinguished from a 
more adventurous conception, we will call “emergence 2”. A characteristics F is 
emergent 2 if A is emergent 1 and F has causal power, which cannot be explained 
by the causal interactions a, b, c . . .. If consciousness is emergent 2, consciousness 
could cause things, which would not be explained by the causal behaviour of the 
neurons.” 

For Minsky, the careful analysis of emergent phenomena “makes generally 
apparent that these phenomena can completely be explained when one take into 
account the interactions between the parts as well as the particularities and 
limitations of the perception and expectations of the observer” [23]. It is the 
precise position of Bertalanffy when he is talking about emergent properties: 
“the knowledge of the set of parts contained in the system and of the relations 
linking them allows to deduce from the behaviour of the parts, the behaviour of 
the whole” [24]. Finally, [25] makes a very interesting contribution to the role of 
non-reductionist concepts in psychology and cognitive science. 

Emergence in Sociology. Without being exhaustive, it is worth mentioning the 
use of the concept of emergence in sociology in particular with the paper of [19] 
and more recently the work of Castelfranchi [26] and of Keith Sawyer [27, 28] 
where important discussions about the various approaches can be found. 

3.1 Towards a Definition of Emergence in MAS 

At the light of the various definitions, in philosophy, in computer science, and in 
cognitive science, it seems essential to come up with a positive, temporal (where 
time appears explicitly) and constructive definition of emergence, especially for 
multi-agent systems. 

The first essential feature of a multi-agent system is that no agent controls 
entirely the dynamics of the population. The agents act only locally and there- 
fore modify the environment by interpreting it given his limited means (using 
the distinctions it is able to make). The agents are limited and there are differ- 
ences of the global system they are unaware of. Therefore, there is an exteriority 
relative to each agent: an environment. The second feature is that the exterior 
of each agent contains other agents. There are several agents in a common envi- 
ronment (they are exterior to each other) . The interpretation of the environment 
by the various agents can possibly be different. In the case of reactive agents, 
the environment contains the objects and the other agents. In the case of cogni- 
tive agents, the environment can also contain messages. Therefore, the dynamics 
proceeds by iteration of interpretation of the local environment by the agents, 
action of the agents on this environment, new interpretation of the modified 
environment and new actions, etc. 
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When such a dynamics (or some of its components) stabilises, we can talk of 
emergence of a structure or of a global function. Notice that at any moment, it is 
the environment possibly modified by all the other agents (and itself) that each 
agent submits to its interpretation. It is the condition for the global dynamics 
to be by more than a simple sum of independent dynamics. In this definition, 
the dynamics of interaction is postulated as a basic condition for emergence of 
phenomena, structures, etc.. Also notice the importance of the link whole-parts, 
which characterises the various kinds of emergent phenomena. 

In the following, we will derive a more operational definition by caracteris- 
ing the whole and the parts, and most importantly the feedback whole-parts. 
This definition is inspired by the preceding definitions and moreover the one by 
S. Forrest. A phenomenon is emergent if and only if we have: 

— a system of entities in interaction whose expression of the states and dynam- 
ics is made in an ontology or theory D ; 

— the production of a phenomenon, which could be a process, a stable state, 
or an invariant, which is necessarily global regarding the system of entities; 

— the interpretation of this global phenomenon either by an observer or by 
the entities themselves via an inscription mechanism in another ontology or 
theory D' . 

The non-linearity of the interactions guarantees the irreducibility of D' to D. In 
other words, D' is not just another way to talk about D. 

The inscription mechanism is essential for two reasons. First, it endows the 
global phenomenon of a causal power (feedback of the level n+ 1 on the level n), 
producing more than a mere epiphenomenon. Second, it allows the global phe- 
nomenon to be perceived and interpreted as a whole by the observer, otherwise 
it would just occur without anybody able to notice it. 

This definition also introduces a distinction depending on the considered 
observers. When the observer is outside of the system, we talk of weak emer- 
gence because the emergence only exists for the external observer. This notion 
of emergence corresponds to the notion of emergence 1 by Searle, or to the first 
order organisation in system theory. As an example of weak emergence, we can 
take the ants that produce a path between a food source and their nest. An ant 
transporting food deposits pheromones. These pheromones diffuse, producing a 
gradient attracting other ants. As a result, a collective back and forth movement 
is produced between the source and the nest as long as there is food available. 
The global phenomenon, i.e. the path, is identified as such by an external ob- 
server (most probably, the ants are not aware of the path itself). The three 
conditions are satisfied: 

— a set of agents interacting in terms of pheromones (and not in terms of 
paths); 

— the production of a stable global phenomenon (as long as there is remaining 
food): i.e. the ants going back and forth between the nest and the food 
source; 
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— the observation of this global phenomenon in terms of a path (geometrically 
speaking) through an inscription, which is the pheromone trace, or more 
easily observable, the ants themselves along the trace. 

Notice that the system is flexible because its behaviour is related to the phero- 
mone gradient and not to the produced path. The path, being a side product, 
can dynamically change depending on new obstacles or change of food source 
position; If the dynamics would depend on the path and then of a representation 
in the ants heads, the decision process to change both the path and its repre- 
sentations would be much heavier. It is this advantage of emergence we would 
like to exploit in emergentist approaches of problem solving and not only the 
intrinsic efficacy of parallelism. 

If the observer is within the system, we obtain a strong emergence , which 
corresponds to the emergence 2 of Searle or to the second order organisation in 
system theory. In this case, the global phenomenon interacts with the entities 
as a whole. As a counter-example, the pheromone path does not interact as 
a path with the ants but only locally by the diffusion mechanism. It would be 
the case if the ants would have a map in their heads. Nevertheless, we have a lot 
of examples of strong emergence in human societies where, for example, the 
produced collective behaviours find explicit representations in the institutions, 
constraining further the individual behaviours. 

From this definition we can derive a number of consequences: 

— the environment plays an important role as an inscription medium. At the 
micro level, it not only provides the ressources for the agents but also the 
interaction medium between the agents. As such, the environment structure 
and properties shape and therefore constrain and coordinate the interactions 
at various time scales depending on the dissipation rate of the substrate (from 
very volatile as the sound to almost permanent like stone buildings). At the 
macro level, as a collective memory recording the historical evolution of the 
individual agent actions and as the inscription medium from which most 
global phenomena can be observed and interpreted at the collective level; 

— any individual agent can participate to several theories D, corresponding to 
multiple points of view it can have, and consequently multiple individual 
roles. But in the case of strong emergence, it can can also react to various 
interpretations D' of the global phenomena it participates to, resulting in 
multiple collective roles; 

— in highly complex system, there is co-existence of both weak and strong 
emergence. It is especially the case in social systems because the collective 
necessarily produces social order, which is out of its awareness, but never- 
theless impacts its functioning. It simultaneously produces representations 
of parts of its global functioning through institutions and more generally 
what Castelfranchi calls cognitive emergence [26]. 
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4 Designing Emergentist Multi-agent Systems 

4.1 Introduction 

Designing multi-agent systems is distinct from designing other types of classical 
computer systems, in particular distributed systems, by at least two dimensions: 

— the importance of the interactions among the agents, which should exceed 
the importance of the agent architecture itself if we want the whole to be 
more than the sum of its constituents; 

— the role of the environment simultaneously as a place of inscription and a set 
of constraints on the multi-agent system dynamics. In distributed systems, 
the environment is often reduced to the communication channels among the 
processes. However, if these channels can be dynamically reconfigured, it is 
essential to introduce spatiality and not only connectivity, in order for the 
connectivity to be deduced from the spatiality and not the converse. 

The design of multi-agent systems raises new challenges and becomes a major 
issue after a somewhat exploratory phase necessary to any new and expanding 
domain. We shall distinguish three types of design approaches: 

— the agent-oriented approaches, which focus on the individual agents and pro- 
pose specification formalisms of their behaviours with various tools (agent- 
oriented programming by [29] or Rao [30], temporal logics approach by 
Wooldridge [31]. These approaches are distinct from pure mono-agent ap- 
proaches by the introduction of communication up to complex negotiation 
protocols; 

— the organisational approaches, which deal with the specification of interac- 
tions through the notions of roles, relations among roles and groups either 
to statically specify interaction networks as in [32] or dynamically as, more 
recently, in [33] and [34]; 

— the emergentist approaches, distinguishing a micro-level of interacting agents 
from a macro- level where the desired global phenomenon is produced, which 
could be either an organisational structure, or the realisation of a task or 
the building of a solution to a problem. These approaches must therefore 
articulate these two levels thanks to a positive definition of emergence as 
proposed earlier and, for example, used in [35, 9]. 

These various approaches are in fact complementary in the sense that the agent- 
oriented approaches specify the entities in interaction, the organisational ap- 
proaches are an important tool to specify what we want to obtain at a macro- 
level, et finally, the emergentist approaches insist on the interactions and on 
the micro/macro articulation, making the link with organisational structures 
(at the global level) using the interactions (at the local level). Separately, they 
suffer limitations because the agent-oriented approaches can hardly take into 
account group dynamics, the direct implementation of organisational structures 
hardly grasps reorganisation dynamics and the emergent approaches still lack 
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sufficiently general methodologies to articulate the micro-level and the macro- 
level. 

To go towards a design methodology of multi-agent systems for problem solv- 
ing by emergence (excluding multi-agent systems for simulation or cooperative 
work as with software agents and CSCW), one can start either from the pro- 
posed definitions of emergence, or from an analysis of existing systems, natural 
or artificial, exhibiting emergent properties in order to determine their common 
features and to deduce a heuristic methodology if not a systematic one. This last 
approach is proposed in this section taking the definition of emergence developed 
earlier as a reading grid. 

After having fixed the vocabulary on problem solving, we shall present a num- 
ber of multi-agent systems obeying the following criteria: 

— their aim is to perform problem solving, and even optimisation; 

— the structure of the state space, as well as the search process, are not explic- 
itly manipulated but emerge from the MAS dynamics ; 

— the solution that results can adapt dynamically to changes of the problem 
data. 

We shall present a synoptic table of their common structure and derive a sys- 
tematic methodology. 

4.2 Problem Solving 

In order to fix the vocabulary, we shall define formally what problem solving is 
(see, for example, [36]). This clarification will allow us to compare more easily 
the classical algorithmic approaches with the multi-agent approaches in general 
and emergentist ones in particular. A problem is specified by a search space E 
constituted by an finite or infinite, discrete (combinatorial) or continuous set of 
states {et}, and a subspace S in E called the space of solutions (or admissible 
states). The problem is dynamic if the state space and/or the solution space 
evolve over time. 

The search space must be described by a set of components, the parametrisa- 
tion and the composition operators allowing to generate the search space (i.e. the 
set of states represented as structures made of these components) . This structure 
is often given as a set of variables v.j and their domain of definition Di. We will 
call S the structure of the search space. 

The set of solutions is generally expressed in intention by a set of constraints. 
When the definition does not allow to directly build one or several solution states, 
we need a search method. One distinguishes two classes of search methods: 

— restriction search consists in reducing the search space by incrementally fix- 
ing the parameters, the components and their composition until one obtains 
a state or a sub-space of the solution set. When one obtains a subspace in 
which no state can be a solution, a backtrack is performed (see, for example, 
CHIP or PrologHI); 
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— repair search consists in building a random state and to modify it whenever 
it does not satisfy the solution criteria (i.e. whenever it is not in the solution 
space) (see [37]). 

Notice that the process goes along a trajectory in the search space and therefore 
can be reformulated as a control problem [38], optimal if we want to optimise 
the search time and stochastic if the search is stochastic. Any combination of 
these two methods is possible. 

One can define on the search space an objective function F and search a 
state that is the best solution state in the sense that it optimises this function F 
(minimum or maximum). In such a case, we have an optimisation problem. 

Classical methods are based on an explicit representation of the current state 
of the search and an algorithm to compute the next state. In our case, we want 
the search process to emerge from interactions among agents in such a way that 
when the system stabilises, the solution can be read as the global state of the 
multi-agent system. It means that the search state representation is distributed 
and also is the computation of the next state. In the following examples, we have 
therefore to identify the structure of the search space, the agents, which have 
been chosen and their interactions. 



4.3 Some Emergentist Multi-agent Systems 

Among the first multi-agent systems to make explicitly reference to the emer- 
gence of a solution to a problem by side-effect of the interaction dynamics, we 
must cite the eco-resolution of Ferber [39] from ethological inspiration. It has 
been applied to a number of problems among which: 

— the blocks world whose problem consists in finding a sequence of executable 
actions to go from an initial configuration to a final configuration. In the 
proposed solution, the agents are the blocks themselves that interact on the 
basis of their relationships. These interactions produce movements whose 
succession will generate a plan and even an executable and acceptable plan. 
In this case, the search space is the space of possible configurations of the 
blocks and the emergent solution is the trajectory in the search space [39]; 

— the magic square consists in putting the tiles in a final configuration by 
moving one tile at a time. In this case, the agents are the tiles that will 
push one another in order to go to their final place and, hence, generate the 
necessary movement sequence [40]. 

This approach is also applied to chess, to show the possibility to have the emer- 
gence of a global strategy from the local interactions of the chess pieces [41]. 

Another kind of emergentist multi-agent systems relies on dynamical systems 
like PACO [42] and SMARPS [43], which have been applied to: 

— contrast line detection in computer vision. The agents are located on the 
pixels of the image and will climb the local intensity gradient, trying to 
keep a given distance from a given number of agents (in general, two) and 
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being influenced by their movements. Lines of agents will eventually stabilise 
on the contrast lines and even follow them if the image changes sufficiently 
slowly with respect to the agent own dynamics. It is one of the rare examples 
where we have an explicit environment, i.e. the image [42]; 

— cartographic generalisation in which the agents are geographic entities (hou- 
ses, trees, road segments, rivers, etc.), which will enter a competition to 
appear on a map of a given resolution. The place occupied by the graphism 
(lines, icons, etc.) will constrain the other agents to move slightly and/or to 
reduce the size and shape of their graphism up to invisibility if there is not 
enough place. The search space is constituted of all the possible positions 
with all the possible graphisms. The actual position and graphical represen- 
tation will emerge from the interactions under the resolution constraint [44] ; 

— production of a set of possible positions of linear structures (roads, power 
lines, etc.) under multi-criteria constraints in land planning. Here, we have 
multiple environments describing the spatial constraints from various points 
of view [43]. 

We can also cite AMROSE [45], a multi-agent system to control an articulated 
robot to sold steel “plaques”. In this case, the rigid parts of the robot are the 
agents trying to go to a certain position without touching the obstacles or the 
other parts of the robot. The result is a trajectory of the end tool of the robot 
resulting from a sequence of commands to the joints. 

We can also mention more applied multi-agent systems using optimisation 
processes as: 

— MARSA is a dynamic scheduling system for flow-shop workplant [46]. This 
system is coupled with a workplant simulator providing various events as: 
command arrivals, the beginning and end of lot treatments, setup and breaks. 
The scheduler provides back the next lots to produce. The agents are the 
commands trying to by produced in accordance with their associated dead- 
lines and the machines trying to minimise their setup time. The interac- 
tions are formulated in terms of allocation (and not of temporal order) with 
an implicit gradient produced by the optimisations to realise. The result is 
a schedule of the commands as produced for a production campaign. Similar 
systems have been deployed by Daewoo in Corea [47] and in other industrial 
applications [48]; 

— AMACOIA is a design system of assembly lines. Given the description of 
a product to assemble and contract cycle time (time between the finishing of 
two parts), the system computes a functional description of an assembly line 
with minimal cost. Two multi-agent systems are coupled: one to explore the 
space of assembly sequences taking into account the product constraints, 
the other one to explore the space of possible assembly lines taking into 
account the production constraints. In the first system, the agents are the 
links between the sub-assemblies (and not the sub-assemblies themselves) 
trying to place themselves into the assembly sequence. In the second system, 
the agents are the operations instantiating the movement axes competing to 
be placed on assembly posts, themselves into cells, and the latter into the 
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assembly line. In each case, the dynamics is in term of assignment, but the 
result is respectively a temporal and a topological structure. 

These cases easily allow to see how emergence can provide a better adaptivity 
of the multi-agent system as a whole. Now we shall make the synthesis of these 
examples on the basis of the definition of emergence and problem solving as 
defined earlier. 



4.4 Towards a Methodology 

The definition of emergence suggests a number of steps to defined a multi-agent 
system: 

— the formal description of the global phenomenon the multi-agent system 
must realise; 

— the projection of this global phenomenon on the interaction structure at 
the micro-level to determine the identities of the agents and the interaction 
dynamics; 

— the specification of individual behaviours of the agents to produce the inter- 
actions generating the global phenomenon we want to observe. 

The lack of direct connection between the macro and the micro levels calls for 
validation tools to guarantee effectively the emergence of the desired global be- 
haviour. This methodology has been entirely developed in [35] and [49]. 

The preceding examples allows to detail further this methodology. In effect, 
all the case have been described as search processes of a solution in search space. 
The structure S of the search space can be either spatial as a path, a map 
or contrast lines, or relational as a schedule or any relational configuration as 
a logical proof considered as a deduction relation on formulas, or spatio-temporal 
as the tasks to be performed by a collective of robots [35]. This structure is 
a composition of elementary entities C: 

Spatial: pheromones deposits, graphisms, proximity links; 

Relational: relations, assignments; 

Spatio-temporal: movements, force applications, etc.. 

In classical programming, we would represent a state of the search space as a data 
structure to be built and modified by a given algorithm. This algorithm would 
depend implicitly or explicitly on all the constraints, initial hypotheses and state 
to elaborate its solution. The problem has to be closed. Any modification would 
require to stop computing or even to change the algorithm itself. If the initial 
data change over time and that new constraints are added or removed dynam- 
ically, the approach becomes extremely difficult to solve because the problem 
becomes open. 

In the multi-agent systems we just described, the state is not explicitly ma- 
nipulated by the agents. The agents interact with one another and with the 
environment in a way, which is indirectly related to the state we want to manip- 
ulate. For example, the agents “task” or “command” are seeking to be placed 
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Table 1 . Comparisons between emergentist cases 



MAS 


Structure 


Components 


Agents 


Interactions 


Ants 


Path 


Pheromone 

deposits 


Ants 


Pheromones 


Termites 


Nest 


Ground 

deposits 


Termites 


Pheromones 


Allocation 


Global 

assignment 


Individual 

assignment 


Tasks and 
resources 


Availability 


MARSA 


Total 

order 


Temporal 

position 


Commands 
and machines 


Deadline and 
setup time 


AMACOIA 


Assembly 

sequence 


Relative 

position 


Link 


Product 

constraints 




Assembly 

line 


Posts 


Operations 


Production 


Blocks 

world 


Configuration 


Block 

relations 


Blocks 


Freedom 


Magic 

square 


Configuration 


Tile 

positions 


Tiles 


Freedom 


PACO 


Contrast 

lines 


Points 

positions 


Points 


Intensity 




Map 


Positions and 
graphisms 


Geographic 

entities 


Freedom 


AMROSE 


Trajectory 


Parts 

positions 


Parts 


Gravity, 

obstacles 



with given criterion as the availability of the resource and their deadline and 
not to placed before or after another one. Therefore, the schedule is only an 
indirect outcome of these interactions. The ants follow the pheromone gradients 
and bring food. The resulting pheromone deposits, which constitute the path are 
only an indirect effect that feedbacks locally on the ant behaviours. In AMA- 
COIA, the agents are the links between the parts when the result is the order 
on the operation. In a similar way, the assembly line is a structure imposed 
on the tools, stations and cells and not the tools, stations and cells themselves 
(which are the agents). The same rationale can be seen in the case of AMROSE, 
where the agents are the rigid parts and the joint angles are resulting from their 
positioning. 

We have systematically reported these observations on the described multi- 
agent systems in the table 1. The first column is the search space structure S , 
the second are the components C, the third column exhibits the chosen agents, 
and the last one on what they interact. 

We would like to comment further on the role of the environment and the 
distinction to make between agents and processes: 

1. we have two distinct roles of the environment in the described systems. One 
is to contain the state of the search during the solving process such that 
it can indirectly feedback on the multi-agent system dynamics. The other 
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is to provide exogeneous constraints on this dynamics as the time line in 
the scheduling systems, the obstacles in the ants paths and, more gener- 
ally, restrictions on the search space. Notice that the agents are themselves 
situated in the environment and this “physical” presence is also source of 
interaction and constraints. For example, in the blocks world or the magic 
square interactions take place because the agents are in the way of other 
agents; 

2. We have to know whether we have to really specify an agent or just a dy- 
namical process taking place in the environment (because the environment 
can have his own dynamics) . It is enough to refer to the definitions of agents 
(as, for example in [4]), which insist on the autonomy of the agents in the 
form or more or less explicit representation of their goals. In the case of 
problem solving, it is enough to identify the choice points, i.e. the compo- 
nents of the search space structure, which allows to potentially explore the 
whole search space (it does not mean we will have to do it exhaustively). 
These choices must result from the interactions among the agents. The rest 
of the search space structure can be processes propagating the consequences 
of theses choices if necessary. For example, in dynamic scheduling the choice 
is a date of production. Recomputing the dates of the other jobs placed be- 
fore or after is just the computation of the consequences of this choice and 
should not be taken in charge by an agent. In the case of simulation we are 
not talking about in this paper, what is a process and what is an agent is 
less clear and is more related to the interpretation of the observer of the 
system [50]. 

We are now at the position to propose a methodology derived from this analysis: 

1. to specify the search space and the structure of its possible states S\ 

2. to determine the elementary components C from which the states of the 
search space are made and among them the choice points determining the 
change from a state to another and potentially guaranteeing the exhaustive 
search through the search space (ergodicity condition); 

3. to determine the entities whose interaction will produce these components. 
We obtain the agents of the system as a kind of negative image of the struc- 
ture to produce; 

4. to determine the objectives and the dynamics of these entities allowing to 
go through the search space. We obtain the production mechanisms of the 
interactions, which will go through the search space and possibly converge 
towards a solution by side-effect ; 

5. to determine the exogeneous constraints guiding the trajectory and to po- 
tentially forbid some parts of the search space. This allows to completely 
define the environment with the inscription of the search space ; 

6. to determine the processes propagating the consequences of the actions of 
the agents, defining the dynamics of the environment itself. It is also possible 
that the exogeneous constraints are directly linked to the world external to 
the multi-agent system (for example in the workplant scheduling, to the 
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workplant itself), in which case the proper dynamics becomes more than 
a simple reaction to the agents interactions; 

7. to validate the design either experimentally if the multi-agent system is 
too complex. For example, by exploration of the topology of the phase 
space as in [11] or theoretically by using either non-linear dynamics, Markov 
chains [51] or statistical dynamics. 

This methodology demonstrates clearly why a multi-agent system is more adapt- 
able and flexible than a classical algorithm. In effect, the solution is not computed 
explicitly by the multi-agent system but emerges from the interactions among 
the agents, which are in dynamical relationship with the problem data and the 
constraints on the possible solutions. This dynamic formulation of accounting 
for the data and the constraints allows the multi-agent system to react spon- 
taneously to the modifications either while trying to find a solution, or after a 
solution has already been found. In reference to the section on problem solving, 
we are in a logic of search by repair and therefore in the paradigm of control. 
In effect, a solution appears as an invariant or a stable state of the dynamics 
of the multi-agent system. However, this formulation raises the problem of the 
observability of the stable state, which represents the solution we are looking 
for. It is the reason why the notion of emergence puts forward the notion of 
observer. We shall detail two reasons to this: 

— the multi-agent system may not know that he found the solution (as it is the 
case for contrast lines), in the sense that no single agent can locally decide 
it but only a global observer of the system. 

— The medium and the inscription process takes all its importance because it 
is this way the observer will be able to observe the solution (as the drawing 
of the links among the agents on the image to visualise the contrast lines). 
The inscription process can also constitute a discretisation process, allowing 
to observe a stable state where the multi-agent with a continuous dynamics 
is in a chaotic attractor. 

This last remark justifies the conclusion of the part on the notion of emer- 
gence [21] that suggests that a theory of emergence has to use a theory of in- 
scription and interpretation and, therefore, calls for semiotic thinking. 

5 Conclusion 

From the definition of complex systems and emergence, we have provided ar- 
gumentations for using multi-agent systems for modelling and designing the 
complex systems, the current computer systems are becoming to be. More than 
a decade of research and developpement of multi-agent problem solving using 
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emergentist approaches give us enough experience to be able to outline a method- 
ology for the conception of such systems. The methodology presented in this 
paper relies directly on the definition of emergence as proposed. Surprisingly 
enough, it is a top-down design methodology of an emergentist system, while 
emergence is intuitively associated to bottom-up production of results. 

Few other methodologies exist to systematically build emergentist multi- 
agent systems. We have to cite in particular the ADELFE [52] approach based on 
the AMAS theory [53] , which is not based directly on the definition of emergence 
and is typically a bottom-up approach. The idea is to specify what each agent 
should locally not do (i.e. be non-cooperative), the correct global behaviour 
emerging accordingly. 

This approach raises a number of perspectives: 

— there is a clear need for software engineering tools to support the design 
process as it exists for ADELFE; 

— the multiplicity of points of view both at the local ( D ) and the global (D') 
levels suggests a strong relationship among roles, norms, institutions and 
so on which is until now situated in more cognitive and agent oriented ap- 
proaches of multi-agent systems. This direction should be further exploited 
with the perspective to link cognitive specifications with actual implemen- 
tations; 

— the problem of validation of emergentist multi-agent systems is very im- 
portant because a part of the achieved flexibility makes it hard to control. 
Two directions could be pursued: either an incremental process through ex- 
ploratory simulation and correction, or formal approaches coming from dy- 
namical systems and statistical mechanics. Adaptive systems could also be 
foreseen with an interaction between the local behaviours and an assess- 
ment of the distance between the actual global behaviour and the expected 
outcome. It is already partially the case in optimising systems when the 
optimality criterion remains global. 
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Abstract. In this paper we discuss some of the ways social order is maintained 
in animal and human realms, with the goal of enriching our thinking about 
mechanisms that might be employed in developing similar means of ordering 
communities of agents. We present examples from our current work in human- 
agent teamwork, and we speculate about some new directions this kind of 
research might take. Since communities also need to change over time to cope 
with changing circumstances, we also speculate on means that regulatory bodies 
can use to adapt. 



1 Introduction 

As computational systems with increasing autonomy interact with humans in more 
complex ways — and with the welfare of the humans sometimes dependent on the 
conduct of the agents — there is a natural concern that the agents act in ways that are 
acceptable to people [7,51]. In addition to traditional concerns for safety and 
robustness in such systems [12], there are important social aspects relating to 
predictability, control, feedback, order, and naturalness of the interaction that must be 
attended to [8,10,50]. In this paper we investigate just some of the ways social order 
is maintained in animal and human realms (sections 2 and 3), with the goal of 
enriching our thinking about mechanisms that might be employed to enhance order in 
mixed human-agent teams. 1 We present examples of such systems that have been 
created to support agent-based applications (section 4), and we speculate about new 
directions this kind of research might take (section 5). Since enduring communities 
also need to change over time to cope with changing circumstances, we speculate 
briefly on means that regulatory bodies can utilize for supporting adaptation (section 
6). Finally, we present some concluding observations (section 7). 



1 In this sense, we agree with the conjecture of Nonnan: “Technology recapitulates 
phylogeny” [50, p. 134], 
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2 Some Sources of Order in the Animal World 

We start by examining some of the ways that animals cooperate and maintain order. 
Why would individuals ever choose to cooperate with others to pursue their aims, 
rather than “going it alone”? In the animal realm, ethnologists and evolutionary 
biologists have taken a fairly common stance with regard to this question. Speaking of 
the process of mutual “attunement” (roughly, “getting to know one another”) among 
individuals, a component process of cooperation, biologist W.J. Smith states: 

Such attunement is necessary when no single individual can fully control an 
encounter — when participants in encounters must depend on each other for a 
useful outcome. The value of that outcome need not be equal for each 
participant, but it must exceed for each the average payoff that would come 
from eschewing the interaction [61, p. 366], 

Smith goes on to discuss two main benefits that accrue from such processes of 
cooperation or “joint activity.” The first is that certain tasks get accomplished that 
could not have been accomplished by any individual. The second is that these kinds of 
activities, over time, yield increased inter-predictability’ among the parties; they come 
to know each other's ways. This can have constructive benefits: for instance, 
knowledge of the other's capabilities might be tapped during future cooperation. It can 
also yield protective benefits: for example, learning the other's “hot buttons” that tend 
to invoke hostility. But the main benefit of predictability is the social order it 
contributes to the group. Gross, mutual unpredictability is almost definitional of 
disorder. Predictability and order are so important to animals that they seem to go to 
great lengths to build but also maintain it: For instance: 

[Some male birds] remember how to recognize previous neighbors by their 
individually distinctive songs and remember the location in which each 
neighbor belongs. Relationships with known neighbors are valuable and those 
with strangers are problematic. Known mutual boundaries can be 
reestablished with much less effort and uncertainty’ than goes into the task of 
working out relationships with new neighbors [61, p. 365], 

Animals engage in joint activities, in which they get to know each other, in part 
through processes of signaling and display that are associated with predictable kinds 
of behaviors. That is, display and signaling behavior among animals supports joint 
activity by providing more or less rough clues to others concerning what each 
individual is about to do. Displays and signals can range widely in form (e.g., 
vocalizations, body posture, facial expressions): 

Each individual has a repertoire of behavior made up of all the many kinds of 
acts it can perform. It can be thought of as continuously choosing among these 
acts, even at times when its behavior is unchanging (among the choices 
available at any instant is to do whatever was done in the previous instant). 

Any choice can be called a ‘behavioral selection. ' 

Each kind of display has a consistent and specifiable relationship to certain 
choices. It is performed in correlation with some kinds of behavior and not 
others. Thus, to know that an individual is performing a particular display is 
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to learn something about the behavior it may select — every display can thus be 
described as encoding messages about behavioral selections [60, p. 87], 

Hence, display behavior has an anticipatory, predictive (but only a probabilistically 
predictive) function. It is a clue, sometimes highly indicative, sometimes much less 
so, 2 to what an individual is about to do. It also decouples actual action from a kind of 
notice that it is about to happen. 3 This decoupling both invites and enables others to 
participate in coordination, support, or avoidance with respect to what might occur. 
This joint engagement in an activity would not be possible if the activity were merely 
executed and not signaled in advance. In this sense, display is an important ingredient 
in enabling things like coordination and teamwork. 

While signaling and display can take many and complicated forms, even in the 
animal world, biologist Smith has advanced ten signal-behavior couplings that appear 
to be pervasive in almost all vertebrates, although they might manifest different 
physical forms in different species [60, pp. 87-126]. The fact that these are so 
pervasive suggests they may be particularly fundamental. We will briefly describe 
each of these types of displays and signals along with possible functions they could 
serve within agent communities. 

2.1 Interactional Displays 

Interactional displays indicate availability or unavailability to participate in joint 
activity. These displays “primarily provide information about the communicator's 
readiness or lack of readiness, to join in acts that involve other individuals” [60, p. 
88]. Since they may be associated with more than one kind of interaction, they do not 
specify any one kind. They might indicate readiness to copulate, associate, attack an 
intruder, and so forth. Hence, they are anticipatory to various kinds of intended joint 
activity, simply signaling a readiness (or lack thereof) to join in association with 
others. 

This category also includes displays indicating absence of opportunity to interact. 
These displays essentially signal that an individual is alone and has nobody else to 
interact with, for example, when an individual is the last remaining at the nest or 
territory. This category also includes signals of shunning interaction. These are 
simply signals that the initiator does not want interaction with others, and this 
intention can range from mild to fierce. 

Example Interactional Forms. Kinds of chirping. Various forms of bowing. 
“Tidbitting” — offering a morsel of food. Forms of touching. Signals from a 
subordinate to a dominant, the purpose of which is to test the dominant's willingness 
to interact, to tolerate interaction. 



2 Sometimes the ambiguity of the signal itself serves an important function, for example as an 
indicator that the signaler’s next move may depend on the response its current move evokes. 

3 To see why this may be useful, consider the signaling functions of the lights on the back of a 
car: “[W]e use turn signals and brake lights to tell others of our actions and intentions. In the 
case of brake lights, we signal actions as we carry them out. In the case of turn signals, we 
signal our intentions before we actually commit them into action. In either case, we allow 
others to know our future actions so that we can ensure that there is no conflict” [50, p. 129]. 
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Absence of opportunity. Loud sounds, loud singing, howling (e.g., one jackal howls, 
and all the rest in the area howl in response), assuming high, visible physical 
positions, special kinds of flight patterns or displays. 

Shunning-. Interestingly, various forms of displaying the tongue. Chittering barks. 
Vocalizations at special, unusual frequencies. 

Possible Functions in Agent Communities. Displays in this general category clearly 
have benefits for coordination among groups of agents by providing information 
about which are or are not in a position to interact with others, in what ways, when, 
and so forth, e.g.: Call me. I am open for calls. I need to talk to someone. May I 
interject, may I say something? 

Absence of opportunity. I am out of touch. I am working all alone. I have no help. I 
have lost contact with everybody. 

Shunning-. Do not attempt to communicate with me for whatever reason, e.g., my line 
is bugged, or 1 am involved in something that cannot be interrupted. Leave me alone. 

While the general interactional displays just discussed are non-specific in the 
activity they portend, others are more specific. 

2.2 Seeking Displays 

Displays indicating that one is seeking joint activity are similar to the interactional 
ones in that they indicate a readiness to participate in some kind joint activity but 
differ in that they indicate active attempt at engaging in a particular kind of activity 
rather than just a general state of availability or receptiveness: 

“ Animals may display while seeking the opportunity to perform some kind of 
activity during what ethnologists call ‘ appetitive ' behavior as distinguished 
from ‘consummatory' behavior in which activity is completed. The behavioral 
selection about which a display provides information if it is done only in this 
way can be termed ‘seeking. ' What a communicator is seeking to do is 
encoded in the same display by a second behavioral selection message. The 
display is interpreted as providing not just the information that a 
communicator is ready to do this second selection, but that its behavior 
includes seeking or preparing to seek an opportunity” [60, p. 118]. 

The seeking display can be associated with many kinds of activities, seeking, for 
example, to interact, associate, copulate, attack, or escape. 

Example Forms. These are associated with so many kinds of behaviors that their 
particular forms vary widely. 

Possible Functions in Agent Communities. Agents that indicate to others what they 
are trying to do can elicit the right form of aid from others, can contribute to possible 
coordination among tasks, and the like. 

2.3 Receptiveness Displays 

Displays indicating receptiveness are the inverse of seeking displays, i.e., they 
indicate a specific response to the seeking of particular kinds of activities by others: 
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“Some displays indicate the behavioral selections that a communicator will 
accept not those it is prepared to perform. At least two behavioral selection 
messages must be provided by such a display, one indicating that the 
communicator will behave receptively and another indicating the class of acts 
to which it is receptive. Effectively, the communicator adopts the role of 
soliciting acts from another individual; it does not offer them ” [60, p. 122], 

The display indicating receptiveness indicates that the communicator is willing to 
engage in a behavior, or set of behaviors, initiated by another. An interesting form of 
soliciting has to do with being receptive to “aid or care” and is common among 
infants who indicate receptiveness to feeding, grooming, shading, and so forth. 
Although often associated with the young, these displays sometimes carry over into 
adult relationships, as when a female mate solicits various forms of “help with the 
nest” from her male partner [60, p. 125]. 

Example Forms. As with seeking displays, receptivity displays are so diverse that 
they defy general description. 

Possible Functions in Agent Communities. As with the seeking displays, receptivity 
displays can contribute to cooperation in the conduct of activity and to the 
coordination among activities. 

2.4 Attack and Escape Displays 

Displays indicating attack and escape : 

“are said to encode either, or both, of attack and escape messages when all 
their occurrence is correlated with a range of attack- or escape-related 
behavior. Behavioral indices of attack differ among and within species, but 
include acts that, if completed, will harm another individual. Escape behavior 
can be any appropriate form of avoidance, ranging from headlong feeing to 
turning aside, or even freezing and other ways of hiding” [60, p. 93], 

Attack and escape displays may differ, but they are sometimes more or less the same 
display, differing only in degree or subtle nuance. They have value both between and 
within groups, for instance, to muster help against an intruder or to avoid inadvertent 
flare-ups (e.g., one group member coming upon another by surprise). Various 
choreographies of interactive displays relating to attacking and escaping can more 
often than not serve to avoid actual combat. Actual fighting is more likely to happen 
among relatively unfamiliar groups [60, p. 94], partly because they have less mutual 
predictability, including prediction of each other's reaction to display activities that 
can fend off real fighting. 

Example Forms. Body posture and orientation. Head bobbing. Forms of jumping. 
Baring teeth. 

Possible Functions in Agent Communities. Displays in this category have increased 
importance when agents are acting in adversarial environments, such as those found 
in military or information intelligence applications. They can be used on the one hand 
to frighten or warn, or on the other hand to signal defeat or flight. 
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2.5 Copulation Behavior Displays 

There are displays indicating copulation behavior. 

“Some displays are performed only before or during the social interactions in 
which eggs are fertilized. These interactions involve either copulation or some 
behavioral analogue such as the amplexus behavior of frogs ” [60, p. 97J. 

Possible Function in Agent Communities. This class of social display would seem 
to have little to do with agents — at least at their current stage of development. 
However, analogues to these displays may be pertinent when certain forms of intricate 
inter-coordination are occurring among agents, involving the need for complex 
cooperation and coordination to carry out the task successfully, e.g., exchanging 
ontologies. The copulatory displays are, after all, cues to the parties involved in 
complex, interdependent operations designed to get an important job done. In a 
simple fashion, a Palm PDA demonstrates this kind of display when it beeps and 
lights up after successful docking in its cradle. 

2.6 Association Maintenance Displays 

There are displays associated with maintaining, staying-in association: 

“Some displays correlate with the behavior involved in remaining with 
another individual. When individuals so associate they remain together 
because one, both, or all will follow, will not leave when the other may not 
follow, and because each permits the others to be nearby.... These displays 
are not common when animals can maintain their association with ease, but 
are used primarily when other behavior may disrupt the group. For instance, 
disruption may result when an individual has just attacked a companion, or 
fees from an approaching predator before the rest of the group reacts, or 
even when an individual that has been absent approaches to resume peaceful 
associating with the group... or when an individual [is] about to move some 
distance from its group in seeking another foraging site, or by an animal able 
to maintain contact with its associates only auditorily” [60, p. 104]. 

These displays appear to provide a kind of reassurance to other group members that, 
despite some possible indications to the contrary, the individual has not broken ranks 
with the group. Such assurances are particularly useful when salient events may raise 
doubt about the continued association. For example, “the likelihood that a group will 
remain together after one or more have fought with each other or with outsiders can 
also be increased by displays encoding an association message” [60, p. 104]. 
Activities, such as foraging or other societal maintenance activities that require an 
emphasis on individual effort and perhaps separation from the group, are also 
prominently associated with association displays. For example, mates who are about 
to be separated for some time exhibit association displays upon leaving and maintain 
these messages during the period of separation to the extent possible (e.g. by special 
vocalizations or gestures — ’’kissing good-bye and calling home every night,” so to 
speak) [60, p. 104]. 




Social Order and Adaptability in Animal and Human Cultures 27 



Example Forms. Special (often oblique) body orientations toward group members. 
Various kinds of vocalizations — clearly, signals that can operate over a distance are 
important in this function. 

Possible Functions in Agent Communities. Ways of indicating allegiance to the 
team and team goals would seem to have a useful place in agent groups and 
teamwork, especially when some agent is temporarily stepping beyond normal bounds 
of location or activity for whatever reason. When agents exercise physical mobility, 
the reporting of location, continued association, and continued commitment to group 
intent would seem to have a potentially beneficial role. 

2.7 Indecisiveness Displays 

Displays indicating indecisiveness signal that the individual is in a state of indecision 
about what to do next. Indicators of indecision are various, ranging from simply 
adopting a static, frozen stance, as if waiting for the situation to provide greater cues, 
to variations on displays that usually indicate action but are modified to increase the 
range of choice. An example of the latter would be moving back-and-forth laterally 
(‘‘pacing'’) with respect to a pertinent stimulus, as opposed to approaching it or 
backing away. Displays for indecisiveness can include behaviors irrelevant and 
inappropriate to the situation, e.g., suddenly, unexpectantly initiating grooming or 
eating [60, p. 107]. 

Example Forms. Irrelevant behavior. Moving back-and-forth laterally in relation to a 
stimulus. 

Possible Functions in Agent Communities. It may be useful for an agent to signal 
that it does not know what to make of some situation, that it is “confused,” or cannot 
figure out what to do next as a means for eliciting help from humans or other agents. 

2.8 Focomotion Displays 

Displays indicating locomotion simply signal that the animal is moving or is about to 
move: 

“[These] displays provide information about a communicator's use of flight 
(or other locomotive) behavior, but not about functional categories of flight 
such as approach, withdrawal, attack, or foraging. The displays correlate with 
all these acts and more. ..some [animals] extend the performance of the 
displays to correlate with hopping or running when they forage on the ground. 
Thus the behavior is viewed as 'locomoting' rather than as flying...'” [60, p. 
108]. 

Example Forms. These displays appear to consist primarily of various forms of 
vocalizations. However, signals indicating that an animal is about to move can be 
more diverse, for example, dances in honeybees, head-tossing in geese. 

Possible Functions in Agent Communities. Signals that indicate that an agent is 
moving or is about to move would seem particularly germane in teams containing 
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mobile agents. As an example of such a display as a warning, think of the distinctive 
sound that large trucks make when they are about to move in reverse. 

2.9 Staying-Put Displays 

Displays indicating remaining with a site are the opposite of the locomotion displays: 

“ Displays performed only when a communicator is remaining at a fixed site 
encode the information he will remain at a single point, in the vicinity of such 
a locus, or in an area that allows considerable movement within fixed 
boundaries. The behavioral selection referred to is simply "staying-put, ’’ 
defined with respect to a site’’ [60, p. 115]. 

Example Forms. Song vocalizations, in particular, are associated with remaining in a 
territory. Birds that do not sing can have special vocalizations for remaining in place, 
e.g., the “ecstatic” vocalization of the Adelie penguin [60, p. 115]. Also included are 
wing-beating, and various specialized postures and movements. 

Possible Functions in Agent Communities. As with displays of locomotion, 
displays of “staying put” are pertinent to mobile agents. 

2.10 Attentiveness Displays 

Displays indicating attentiveness to a stimulus simply convey that the communicator 
is attending to something and monitoring it. 

Example Forms. Three distinct barks of a prairie-dog, indicating three different 
phases of monitoring. Such barks might indicate, for example, that a predator is in the 
vicinity. 

Possible Functions in Agent Communities. For agents, these signals could portend 
that something important might be happening. It would be useful in agent 
communities to have a general indicator of alert, that something significant might be 
transpiring at a particular location or involving a particular agent. Appropriate 
response, of course, would require additional information. In the animal world, for 
instance, this additional information sometimes indicates the location of the stimulus. 



3 Some Sources of Order in the Human World 

It is not surprising that joint activity — and the “getting to know each other” both 
necessary for it and engendered by it — are also important to humans. Additionally, 
many of the same benefits are accrued — in particular, inter-predictability and its 
relationship to coordination and an orderly society. Moreover, the same basic 
components are involved, including signals understood by both parties to be an 
invitation to engage in joint activity. However, because of our wider behavioral 
repertoire, the greater complexity of our communication processes, and our reduced 




Social Order and Adaptability in Animal and Human Cultures 29 



dependence on biological determinism, human cooperation and regulatory processes 
take on an even greater variety of forms. 4 

In fact, because of our vast behavioral repertoire, and because we are so 
underdetermined in our biology, the argument has been made that a very large portion 
of what humans do and create is constituted to “control ourselves”! In this view, the 
role of human culture is that of avast, fabricated self-regulatory mechanism [29] : 5 

I want to propose two ideas: The first of these is that culture is best seen not 
as complexes of concrete behavior patterns — customs, usages, traditions, 
habit clusters — as has, by and large, been the case up to now, but as a set of 
control mechanisms — plans, recipes, rules, instructions (what computer 
engineers call 'programs')— for the governing of behavior. The second idea is 
that man is precisely the animal most desperately dependent upon such 
extragenetic, outside-the-skin mechanisms, such cultural programs, for 
ordering his behavior... (p. 44) 

Man is in need of such symbolic sources of illumination [i.e., human-created 
cultural control mechanisms — addition ours] to find his bearings in the world 
because the non-symbolic sort that are constitutionally engrained in his body 
cast such a diffuse light. The behavior patterns of lower animals are, at least 
to much greater extent, given to them with their physical structure: genetic 
sources or information order their actions within much narrower ranges of 
variation, the narrower and more thouroughgoing the lower the animal. For 
man, what are innately given are extremely general response capacities, 
which, although they make possible far greater plasticity, complexity, and, on 
the scattered occasions when everything works as it should, effectiveness of 
behavior, leave it much less precisely regulated. This then, is the second face 
of our argument: Unregulated by cultural patterns — organized systems of 
significant symbols — man 's behavior would be virtually ungovernable, a mere 
chaos of pointless acts and exploding emotions, his experience virtually 
shapeless. Culture, the accumulated totality of such patterns, is not just an 
ornament of human existence but — the principal basis of its specificity’ — an 
essential condition for it. (pp. 45-46) 

In summary, according to this argument people create and have created cultures and 
social conventions — albeit in many disparate forms across mankind that can be hard 
for outsiders to understand — to provide order and predictability. This is also the main 
reason we claim, following Smith's arguments above that animals cooperate at all, 
when they do — that is, in order to make themselves better known and more 
predictable to each other. Furthermore, it would seem to follow from Geertz's 
argument that the more autonomous the agents involved, the more need there is for 
such regulation and the wider the variety of forms it might take. 

Order and predictability may have a basis in the simple cooperative act between 
two people, in which the parties “contract” to engage together in a set of interlinked, 



4 For a comprehensive and interesting treatment of these kinds of issues regarding joint 
activity in humans, see[15], 

5 We recognize that Geertz represents only one of many views of culture, but a discussion of 
competing views is beyond the scope of this paper. 
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mutually beneficial activities. From this simple base, in humans at least, there are 
constructed elaborate and intricate systems of regulatory tools, from formal legal 
systems, to standards of professional practice, to norms of proper everyday behavior 
(along with associated methods of punishment or even simple forms of shaming for 
violations of these). 



4 The Problem of Adaptability 

While the discussion so far has dealt mainly with the maintenance of order, change is 
also necessary in perpetuating healthy societies, especially if those societies are 
expected to adapt to new circumstances and endure over long periods of time. While 
we cannot investigate adaptation mechanisms in depth in this paper, we feel it 
important to point out that such mechanisms of change are recognized as critical in 
both animal and human societies. 

For instance, while animal and human signals carry a certain core nature and 
meaning in a given community, this meaning is not completely rigid or mechanical, 
and may be very different in different contexts. 6 Such interaction can often best be 
described as a kind of “improvisation” — embodying considerable novelty while 
respecting the rules of the form [53]. 

Take for example two different cases of a human signaling a call to joint activity 
with another, in fact, signaling the same call in the two cases, “help me.” In the first 
instance, the solicitor is sinking in quicksand, and in the other case the solicitor is 
posed at one end of a heavy table that needs moving. The particulars of what might 
ensue will depend on the nature of the two different circumstances but also on the 
particular individuals involved. In the first case the party responding to the request for 
help may try to throw a rope. However, if there were a history of bad will between 
himself and the person in the quicksand, he might also just lay back and watch him 
slowly sink [lack of will]. In the second case, the party responding to the request for 
help might, on the one hand, go to the unmanned end of the table and try to help lift 
(and he would not throw a rope — due to the basic circumstantial difference). On the 
other hand, he might not — his response may depend on how strong he thinks he is or 
if he has sustained an injury (degree of capability) or depending on his personal 
history of experienced helpfulness from the individual making the request [(lack of 
will) — “He never helps me!”] [after[60, p. 224]]. 

Thus the elements of consistency, but also potential novelty, may both be 
necessary to signaling activity in the real world, because the world is never static: 

"In all social events, the behavior of participants must engender considerable 
predictability’. Without predictability, events falter and their orderliness 



6 Nonnan [50, p. 130] gives the following example of this phenomenon from traffic behavior 
in different countries: “In Mexico, one wins by aggression. In Britain, one wins by 
politeness and consideration. [In Mexico, when two cars approach a narrow bridge from 
different directions, flashing your headlights means, ‘I got here first, so keep out of my 
way.’ However] in Britain, in a similar situation, the car that flashes its lights first is 
signaling, ‘I see you, please go ahead and I will wait.’ Imagine what happens when a 
Mexican driver encounters a British driver.” 
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dissipates ... [But] the dilemma addressed in this volume is that development 
of shared signals and codes necessarily leads to conformity’ in signaling, but 
conformity cannot cope well with changing or novel events and, when rigid, is 
maladaptive” [61, p. 366], 

Hence, all signaling must accommodate elements of variation in the pertinent core 
joint activity conveyed by the signal. These variations are sensitive at least to the 
particulars of the circumstances and parties involved; that is, the variation on the core 
activity is context-sensitive: 

“ Another crucially important aspect of all communication is that it is context- 
dependent. That is, although the information made available by a formalized 
signal is largely consistent, the significance or 'meaning' of that information 
in any event, its interpretation by the individual responding to the signal is 
affected by information in addition to the signal. This is another form of 
openness in communication and an important means of dealing with novelty’. 

This requisite ability’ to alter responses to signals as circumstances change is 
also the basis of our calibration of individual signalers. Clues to their 
identities become clues to the specific significances of their signals, although 
only for individuals who are sufficiently familiar with them ” [61, p. 368]. 

With regard to change and adaptation in culture and its regulatory role, modern 
biologists have increasingly emphasized that the natural selection process includes not 
only basic biology but also the equally complex elements of culture, cultural change 
and cultural selection. For instance Mayr has emphasized that “a person is a target of 
selection in three different contexts: as an individual, as a member of a family... and 
as a member of a social group.” [44, p. 251]. The latter two, at least, implicate culture 
in the sense we have been addressing it in this essay. Geertz has gone farther, 
essentially arguing that humanity and culture are so tightly intertwined that the 
human-culture system is the unit of selection [29, p. 67]. In short, in enduring 
societies, culture is not static. 

Although such nuanced tailoring of communication and culture to circumstance 
may not always prove necessary in the working interactions of pure agent teams, the 
need for such tailoring and adjustment will almost surely arise in mixed human-agent 
teams, as their work together becomes increasingly consequential and as they sustain 
their interactions for long periods. This is another key element of making agents 
acceptable to humans. To be acceptable to humans, agents must conform to certain 
standards of predictability, but they also must not exhibit bald, na'ive-looking rigidity. 

While recognizing the importance of adaptation, because of the tremendous 
challenges currently involved in machine learning, our own work has been initially 
focused on understanding and enabling various forms of order in agent communities. 
We will briefly address adaptation again in section 6. 
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Fig. 1 . Required elements of future infrastructure for agents 



5 Building Cultures for Agent Communities: Sources of Order 

Our agent research and development efforts over the past decade have maintained a 
consistent trend. We have been progressively off-loading selected classes of 
knowledge, some aspects of decision-making, and various kinds of specialized 
reasoning and problem solving from individual agents into a common environment 
shared by all agents of a given community, regardless of the nature or sophistication 
of their internals or the platform on which they are running. 7 This has taken the form, 
for instance, of the creation of various types of services and various bodies of policy 
that help regulate conduct across communities of heterogeneous agents running on 
various platforms. It is in this sense that what we have been doing might be thought of 
as creating “cultures” for agent communities, especially communities that might 
endure for long periods of time. We have termed this kind of approach “terraforming 
cyberspace” (referring to the aspect of the effort that aims to make networked 
environments a more habitable place for agents) and “cyberforming terraspace” 
(referring to the aspect of the effort that aims to embed socially-competent agents in 
the physical world) [12]. To support sustainability of groups of agents over long 
periods, we have envisioned basic types of services that will be needed (Fig. 1.). At a 
minimum, future infrastructure must go beyond the bare essentials of support to 
provide pervasive life support services (relying on mechanisms such as orthogonal 
persistence [36] and strong mobility [62,63]) that help ensure the survival of agents 
designed to live for long periods of time. Beyond the basics of individual agent 
protection, these communities will depend on legal services, based on explicit 
policies, to ensure that rights and obligations are monitored and enforced. Benevolent 
social services might also be provided to proactively avoid problems and help agents 
fulfill their obligations. Although some of these elements exist in embryo within 
specific agent systems, their scope and effectiveness has been limited by the lack of 
underlying support at both the platform and application levels. 



7 It could also be said that we have been moving elements from the “sharp end” to the “blunt 
end” of agents’ activity, as these two terms have been characterized by David Woods and 
colleagues [20], 
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In the remainder of this section, we will briefly review efforts to create and 
regulate agent cultures through the use of norms and policies (5.1) 8 . We will discuss 
the relationship between plans and policy (5.2) and between autonomy and policy 
(5.3). We will introduce KAoS (5.4) and some basic categories of technical and social 
policies (5.5). Then we will provide a few examples of policies that address joint 
activity and signaling, that we are developing for military and space applications 
(5.6). 

5.1 Norms and Policy 

In the early 20 th century, a legal theorist named Wesley Newcomb Hohfeld developed 
a theory of fundamental legal concepts [32] from which most of current work on 
theories of normative positions have taken at least some degree of inspiration (see 
e.g., [40,57]). 

The idea of building strong social laws into intelligent systems can be traced at 
least as far back as the 1940s to the science fiction writings of Isaac Asimov [3]. In 
his well-known stories of the succeeding decades he formulated a set of basic laws 
that were built deeply into the positronic-brain circuitry of each robot so that it was 
physically prevented from transgression. Though the laws were simple and few, the 
stories attempted to demonstrate just how difficult they were to apply in various real- 
world situations. In most situations, although the robots usually behaved “logically,” 
they often failed to do the “right” thing — typically because the particular context of 
application required subtle adjustments of judgments on the part of the robot (e.g., 
determining which law took priority in a given situation, or what constituted helpful 
or harmful behavior). 9 

Shoham and Tennenholtz [58] introduced the theme of social laws into the agent 
research community, where investigations have continued under two main headings: 
norms and policies. Drawing on precedents in legal theory, social psychology, social 
philosophy, sociology, and decision theory [71], norm-based approaches have grown 
in popularity [6,21,41,42]. In the multi-agent system research community, Conte and 
Castelfranchi [19] found that norms were variously described as constraints on 



8 We have concentrated first on mechanisms for establishing order and predictability in agent 
communities because at the current state of agent development these seem to be the greatest 
concerns of both producers and consumers of agent technologies. Others have focused on 
issues of “democracy,” micro-economics, and other forms of relative freedom in open 
societies of agents e.g. [14] [45][49] 

9 In an insightful essay, Roger Clarke explores some of the implications of Asimov’s stories 
about the laws of robotics for information technologists [16]. Weld and Etzioni [72] were 
the first to discuss the implications of Asimov’s first law of robotics for agent researchers. 
Like most norm-based approaches described below (and unlike most policy-based 
approaches) the safety conditions are taken into account as part of the agents’ own learning 
and planning processes rather than as part of the infrastructure. In an important response to 
Weld and Etzioni’ s “call to arms,” Pynadath and Tambe [52] develop a hybrid approach that 
marries the agents’ probabilistic reasoning about adjustable autonomy with hard safety 
constraints to generate “policies” governing the actions of agents. The approach assumes a 
set of homogeneous agents, which are motivated to cooperate and follow optimally 
generated policies 
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behavior, ends or goals, or obligations. For the most part, implementations of norms 
in multi-agent systems share three basic features: 

• they are designed offline; or 

• they are learned, adopted, and refined through the purposeful deliberation of each 
agent; and 

• they are enforced by means of incentives and sanctions. 

Interest in policy-based approaches to multi-agent and distributed systems has also 
grown considerably in recent years (http://www.policy-workshop.org) [22,37,67]. 
While sharing much in common with norm-based approaches, policy-based 
perspectives differ in subtle ways. Whereas in everyday English the term norm 
denotes a practice, procedure, or custom regarded as typical or widespread, a policy is 
defined by the American Heritage Online dictionary as a “course of action, guiding 
principle, or procedure considered expedient, prudent, or advantageous.” Thus, in 
contrast to the relatively descriptive basis and self-chosen adoption (or rejection) of 
norms, policies tend to be seen as prescriptive and externally imposed entities. 
Whereas norms in everyday life emerge gradually from group conventions and 
recurrent patterns of interaction, policies are consciously designed and put into and 
out of force at arbitrary times by virtue of explicitly recognized authority. 10 These 
differences are generally reflected in the way most policy-based approaches differ 
from norm-based ones with respect to the three features mentioned above. Policy- 
based approaches: 

• support dynamic runtime policy changes, and not merely static configurations 
determined in advance; 

• work involuntarily with respect to the agents, that is, without requiring the agents 
to consent or even be aware of the policies being enforced; thus aiming to 
guarantee that even the simplest agents can comply with policy; and 

• wherever possible they are enforced preemptively, preventing buggy or malicious 
agents from doing harm in advance rather than rewarding them or imposing 
sanctions on them after the fact. 

5.2 Plans and Policy 

Policy management should not be confused with planning or workflow management, 
which are related but separate functions. Planning mechanisms are generally 
deliberative (i.e., they reason deeply and actively about activities in support of 
complex goals) whereas policy mechanisms tend to be reactive (i.e., concerned with 
simple actions triggered by some environmental event) [27, pp. 161-162]. Whereas 
plans are a unified roadmap for accomplishing some coherent set of objectives, bodies 
of policy collected to govern some sphere of activity are made up of diverse 
constraints imposed by multiple potentially-disjoint stakeholders and enforced by 
mechanisms that are more or less independent from the ones directly involved in 
planning. The independence of policy, reasoning, and enforcement mechanisms from 



10 While it is true that over time norms can be fonnalized into laws, policies are explicit and 
formal by their very nature at the outset. 



Social Order and Adaptability in Animal and Human Cultures 35 



planning capabilities helps assure that, wherever possible, key constraints imposed by 
the humans are respected even in the face of buggy or malicious agents on the one 
hand, and poorly designed or oversimplified plans on the other. Plans tend to be 
strategic and comprehensive, while policies, in our sense, are by nature tactical and 
piecemeal. In short, we might say that while policies constitute the “rules of the 
road” — providing the stop signs, speed limits, and lane markers that serve to 
coordinate traffic and minimize mishaps — they are not sufficient to address the 
problem of “route planning.” 11 

5.3 Autonomy and Policy 12 

Some important dimensions of the relationship between autonomy and policy can be 
straightforwardly characterized by reference to figure 1. 13 

The outermost rectangle, labeled potential actions, represents the set of all actions 
defined in some ontology under current consideration. 14 In other words, it contains the 
union of all actions for all actors currently known to the computational entities that 
are performing reasoning about adjustable autonomy and mixed-initiative interaction. 
Note that there is no requirement that all actions that an agent may take be represented 
in the ontology; only those which are of consequence for policy representation and 
reasoning need be included. 

The rectangle labeled possible actions represents the set of potential actions whose 
achievement by some agent is deemed sufficiently imaginable in the current context. 
Of these possible actions, any given actor 15 (e.g., Agent A) will likely only be deemed 
to be capable of performing some subset. Capability is a function of the abilities and 
resources available to an actor attempting to undertake some action. An actor's ability 
is the sum of its own knowledge and skills, whereas its resources consist of all other 
assets it can currently draw on in the performance of the action. Two actors, Agent A 



n We are exploring the relationship between policy and planning in new research with James 
Allen [2] [9], 

12 More detail on this topic can be found in [9]. 

13 We can make a rough comparison between some of these dimensions and the aspects of 
autonomy described by Falcone and Castelfranchi [25]. Environmental autonomy can be 
expressed in tenns of the possible actions available to the agent — the more the behavior is 
wholly deterministic in the presence of a fixed set of environmental inputs, the smaller the 
range of possible actions available to the agent. The aspect of self-sufficiency in social 
autonomy relates to the ranges of what can be achieved independently vs. in concert with 
others; deontic autonomy corresponds to the range of permissions and obligations that 
govern the agent’s choice among actions. 

14 The tenn ontology is borrowed from the philosophical literature, where it describes a theory 
of what exists. Such an account would typically include tenns and definitions only for the 
very basic and necessary categories of existence. However, the common usage of ontology 
in the knowledge representation community is as a vocabulary of representational tenns and 
their definitions at any level of generality. A computational system’s “ontology” defines 
what exists for the program — in other words, what can be represented by it. 

15 For discussion purposes, we use the tenn actor to refer to either a biological entity (e.g., 
human, animal) or an artificial agent (e.g., software agent, robotic agent). 
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and Agent B, may have both overlapping and unique capabilities . 16 If a set of actors is 
jointly capable of performing some action, it means that it is deemed to be possible 
for it to be performed by relying on the capabilities of both actors. Some actors may 
be capable of performing a given action either individually or jointly; other actors 
may not be so capable. 

In addition to the descriptive axis describing various dimensions of capability, 
there is a prescriptive axis that is defined by policies specifying the various 
permissions and obligations of actors. Authorities may impose or remove involuntary 
policy constraints on the actions of actors. Alternatively, actors may voluntarily enter 
into agreements that mutually bind them to some set of policies so long as the 
agreement is in effect. The effectivity of an individual policy is the set of conditions 
that determine when it is in or out of force. 

The set of permitted actions is defined by authorization policies that specify which 
actions an actor is allowed {positive authorizations or A+ policies) or not allowed 
{negative authorizations or A- policies) to perform in a given context. The intersection 
of what is possible and what is permitted to a given set of actors defines a set of 
available actions. 

Of those actions that are available to a given actor, some subset may be judged to 
be independently achievable by it in the current context. Some actions, on the other 
hand, would only be jointly achievable. 



Potential Actions I Permitted Actions 



Possible Actions 1 Available Actions | Obligated Actions 



Capable of 
Agent A 



Independently 
Achievable 
by Agent A 



Required of 
Agent A, 
Independently 



Capable of 
Agent B 



Independently 
Achievable 
by Agent B 



Required of 
Agent B, 
Independently 



Jointly Capable Jointly Achievable I Jointly Required 
of Agent A and B ( by Agent A and B * of Agent A and B 



Key: ^3 Capability Dimension ^ J Policy Dimension 



Fig. 2. Basic dimensions of adjustable autonomy and mixed-initiative interaction 



16 Note that although we show A and B sharing the same set of possible actions in figure 2, 
this is not necessarily the case. 
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Finally, the set of obligated actions is defined by obligation policies that specify 
actions that an actor is required to perform (positive obligations or 0+ policies) or for 
which such a requirement is waived (negative obligations or O- policies). Positive 
obligations commit the resources of actors, reducing their current overall capability 
accordingly. Jointly obligated actions are those that two or more agents are explicitly 
required to perform. 

A major challenge is to ensure that the degree of autonomy is continuously and 
transparently adjusted to be consistent with explicitly declared policies which 
themselves can, ideally, be imposed and removed at any time as appropriate [48]. For 
example, one goal of the agent or external entity performing such adjustments should 
be to make sure that the range of permissible actions do not exceed the range of those 
that are likely to be achievable by the agent. 17 While the agent is constrained to 
operate within whatever deontic bounds on autonomy are currently enforced as 
authorization and obligation policies, it is otherwise free to act. 

Thus, the coupling of autonomy with policy gives the agent maximum opportunity 
for local adaptation to unforeseen problems and opportunities, while assuring humans 
that agent behavior will be kept within desired bounds. 

In principle, the actual adjustment of an agent's level of autonomy could be 
initiated either by a human, the agent, or some other software component. 18 To the 
extent we can adjust agent autonomy with reasonable dynamism (ideally allowing 
handoffs of control among team members to occur anytime) and with a sufficiently 
fine-grained range of levels, teamwork mechanisms can flexibly renegotiate roles and 
tasks among humans and agents as the situation demands. Such adjustments can also 
be anticipatory when agents are capable of predicting the relevant events [5,25]. 
Research in adaptive function allocation — the dynamic assignment of tasks among 



17 If the range of achievable actions for an agent is found to be too restricted, it can, in 
principle, be increased in any combination of four ways: 1. removal of some portion of the 
environmental constraints, thus increasing the range of possible actions; 2. increasing its 
permissions; 3. making additional external help available to the agent, thus increasing its 
joint capabilities; or 4. reducing an agent’s current set of obligations, thus freeing resources 
for other tasks. Of course, there is a cost in computational complexity to increasing the 
range of actions that must be considered by an agent — hence the judicious use of policy 
where certain actions can either be precluded from consideration or obligated with 
confidence in advance by a third party. 

is Cohen [18] draws a line between those approaches in which the agent itself wholly 
determines the mode of interaction with humans (mixed-initiative) and those where this 
determination is imposed externally (adjustable autonomy). Additionally, mixed-initiative 
systems are considered by Cohen to generally consist of a single user and a single agent. 
However, it is clear that these two approaches are not mutually exclusive and that, in an 
ideal world, agents would be capable of both reasoning about when and how to initiate 
interaction with the human and also of subjecting themselves to the external direction of 
whatever set of explicit authorization and obligation policies were currently in force to 
govern that interaction. Additionally, there is no reason to limit the notion of “mixed 
initiative” systems to the single agent-single human case. Hence we prefer to think of 
mixed-initiative systems as being those systems that are capable of making context- 
appropriate adjustments to their level of social autonomy (i.e., their level or mode of 
engagement with the human), whether a given adjustment is made as a result of reasoning 
internal to the agent or due to externally-imposed policy-based constraints. 
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humans and machines — provides some useful lessons for implementations of 
adjustable autonomy in intelligent systems [31]. 

When evaluating options for adaptively reallocating tasks among team members, it 
must be remembered that dynamic role adjustment comes at a cost. Measures of 
expected utility can be used to evaluate the tradeoffs involved in potentially 
interrupting the ongoing activities of agents and humans in such situations, in order to 
communicate, coordinate, and reallocate responsibilities [18,33,34]. It is also 
important to note that the need for adjustments may cascade in complex fashion: 
interaction may be spread across many potentially-distributed agents and humans who 
act in multiply-connected interaction loops. For this reason, adjustable autonomy may 
involve not merely a shift in roles among a human-agent pair, but rather the 
distribution of dynamic demands across many coordinated actors. 19 Defining explicit 
policies for the transfer of control among team members and for the resultant required 
modifications to coordination constraints can prove useful in managing such 
complexity [54]. Whereas in the past goal adoption and the commitment to join and 
interact in a prescribed manner with a team have sometimes occurred as part of a 
single act in early teamwork formulations, researchers are increasingly realizing the 
advantages of allowing the respective acts of goal adoption, commitment to work 
jointly with a team, and the choice of specific task execution strategies to be handled 
with some degree of independence [4,48]. Over the last few years, we have been 
developing a set of services within a framework called KAoS to accomplish just these 
kind of goals. 

5.4 Overview of KAoS 

KAoS is a collection of componentized agent services compatible with several 
popular agent frameworks, including Nomads [63], the DARPA CoABS Grid [38], 
the DARPA ALP/Ultra*Log Cougaar framework (http://www.cougaar.net), CORBA 
(http://www.omg.org), and Voyager (http://www.recursionsw.com/osi.asp). While 
initially oriented to the dynamic and complex requirements of software and robotic 
agent applications, KAoS services are also being adapted to general-purpose grid 
computing (http://www.gridfomm.org) and Web services 

(http://www.w3.org/2002/ws/) environments as well [35]. 

KAoS domain services provide the capability for groups of agents, people, 
resources, and other entities to be structured into organizations of agent domains and 
subdomains to facilitate agent-agent collaboration and external policy administration. 

KAoS policy services allow for the specification, management, conflict resolution, 
and enforcement of policies within domains. The KAoS Policy Ontologies (KPO), 
represented in OWL [69], distinguishes between authorizations (i.e., constraints that 
permit or forbid some action) and obligations (i.e., constraints that require some 
action to be performed, or else serve to waive such a requirement) [22]. 



19 As Hancock and Scallen [31] rightfully observe, the problem of adaptive function allocation 
is not merely one of efficiency or technical elegance. Economic factors (e.g., can the task be 
more inexpensively perfonned by humans, agents, or some combination?), political and 
culhiral factors (e.g., is it acceptable for agents to perform tasks traditionally assigned to 
humans?), or personal and moral considerations (e.g., is a given task enjoyable and 
challenging vs. boring and mind-numbing for the human?) are also essential considerations. 
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5.5 Technical and Social Policy Categories 

To increase the likelihood of human acceptance of agent technology, successful 
systems must attend to both the technical and social aspects of policy [51]. From a 
technical perspective, we want to be able to help ensure the protection of agent state, 
the viability of agent communities, and the reliability of the resources on which they 
depend [12]. To accomplish this, we must guarantee, insofar as is possible, that the 
autonomy of agents can always be bounded by explicit enforceable policy that can be 
continually adjusted to maximize the agents' effectiveness and safety in both human 
and computational environments. From a social perspective, we want agents to be 
designed to fit well with how people actually work together and otherwise interact. 
Explicit policies governing human-agent interaction, based on careful observation of 
work practice and an understanding of current social science research, can help assure 
that effective and natural coordination, appropriate levels and modalities of feedback, 
and adequate predictability and responsiveness to human control are maintained 
[11,26,50]. These and similar technical and social factors are key to providing the 
reassurance and trust that are the prerequisites to the widespread acceptance of agent 
technology for non-trivial applications. 

We currently classify technical policies into six categories: 

• Authentication policies. This category of policies is concerned with 
assuring that identification of proper users is associated with various agent 
commands and actions. 

• Data and resource access and protection policies. These policies control 
access to resources, information, and services, and specify any constraints 
on data protection (e.g., encryption). 

• Communication policies. Communication policies govern message passing 
among individuals and groups, including forms of content filtering and 
transformation [64,65]. 

• Resource control policies. Going beyond simple access control, these 
policies control the amount and rate of resource usage (e.g., CPU, 
memory, network, hard disk, screen space) [62,63]. 

• Monitoring and response policies. These policies typically represent 
obligations for the system to perform specific monitoring and response 
actions (e.g., logging, response to authorization failures or changes to 
global system defense postures). 

• Mobility policies. Mobility policies govern the physical movement of 
software or hardware agents [39]. 

We are also currently developing social policies within six categories: 20 



20 The motivation for such policies is eloquently expressed by Norman [50, p. 126-127]: “One 
of the reasons that modem technology is so difficult to use is because of [its] silent, invisible 
operation [when compared with mechanical devices]. The videocassette recorder, the digital 
watch, and the microwave oven — none is inherently complicated. The problem for us is 
their lack of communication. They fail to interact gracefully. They demand attention and 
services, but without reciprocating, without providing sufficient background and context. 
There is little or no feedback.... The modern information-processing machine fits the 
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• Organization policies. This category of policies includes those that 
specify relationships among classes of agents, e.g., policies about 
delegation of responsibilities and agent registration in domains. 

• Notification policies. It is important that important information be 
conveyed to the appropriate people at the appropriate time and with an 
appropriate modality. Based on the work of [55,56], we are building 
ontologies supporting policies for categories of agents, roles, 
notifications, latency, focus of attention, and presence as a foundation 
for policies governing context-sensitive notification. 21 

• Conversation policies. Explicit conversation policies simplify the work 
of both the agent and the agent designer [30]. Such policies include, for 
example, constraints on conversation message sequencing, termination 
conditions, and timeouts. 

• Nonverbal expression. These policies govern signaling and display 
behavior of agents. Detailed examples are given below. 

• Collaboration policies. Policies governing team coordination are classed 
in this category, including the formation and discharge of joint goals as 
is central in traditional multi-agent teamwork theory [17,66]. 

• Adjustable autonomy policies. These policies regulate levels of, and 
adjustments to, levels of agent autonomy [23]. 

A fuller discussion of examples from each of these categories may be found in [7]. 

We now discuss a few simple examples of policy relating to the theme of display 
and signaling behavior. The policy examples are drawn from our studies of the 
Personal Satellite Assistant (PSA), currently under development at NASA Ames. The 
PSA is a softball-sized flying robot, designed to help astronauts, that is being 
developed to operate onboard spacecraft in pressurized micro-gravity environments 
[1,10,24,28,59]. 

For clarity's sake, we will present example policies in ordinary English rather than 
in DAML. For brevity's sake, the policies will be presented in an incomplete form. 
Each example is preceded by A+, A-, 0+, or O- to indicate whether it is respectively a 
positive authorization, negative authorization, positive obligation, or negative 
obligation. 



stereotype of an antisocial, technological nerd. It works efficiently, quietly, and 
autonomously, and it prefers to avoid [social] interactions with the people around it.”. 

21 Of course the important point in context-sensitive notification in our day of information and 
sensory overload is sometimes not helping the infonnation get through but rather blocking, 
filtering, or rechanneling it in appropriate ways: “Most instrument panels [use lights, 
buzzers, and alarms to] tell us when something is wrong and needs immediate attention. No 
social protocols, no etiquette. No checking to see whether we are busy at some other 
activity, usually not even a check to see if other alarms or warnings are also active. As a 
result, all the alanns and warnings scream in their self-centered way... In places that have 
large control panels,... the first act of the human operators is to shut off the alarms so they 
can concentrate upon the problem” [50, p. 128], 
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5.6 Nonverbal Expression Policy: Examples 

Where possible, agents usually take advantage of explicit verbal channels for 
communication in order to reduce the need for relying on current primitive robotic 
vision and auditory sensing capabilities [47, p. 295]. On the other hand, animals and 
humans often rely on visual and auditory signals in place of explicit verbal 
communication for many aspects of coordinated activity. As part of our work on 
human-robotic interaction for NASA, the Army, and the Navy, we are developing 
policies to govern various nonverbal forms of expression in robotic and software 
agents. These nonverbal behaviors will be designed to express not only the current 
state of the agent but also — importantly — to provide rough clues about what it is 
going to do next. 

Maes and her colleagues were among the first to explore this possibility in her 
research on software agents that continuously communicated their internal state to the 
user via facial expressions (e.g., thinking, working, suggestion, unsure, pleased, and 
confused) [43]. Breazeal has taken inspiration from research in child psychology [68] 
to develop robot displays that reflect four basic classes of preverbal social responses: 
affective (changing facial expressions), exploratory (visual search, maintenance of 
mutual regard with human), protective (turning head away), and regulatory 
(expressive feedback to gain caregiver attention, cyclic waxing and waning of internal 
states, habituation, and signals of internal motivation) [13]. Norman has investigated 
the role of signaling, not only in effective coordination and communication, including 
the communication of emotion, but also with regard to the role of deception in human 
and agent affairs [50]. Books on human etiquette [70] contain many descriptions of 
appropriate behavior in a wide variety of social settings. Finally, in addition to this 
previous work, we think that display and signaling behavior among people [46] and 
groups of animals will be one of the most fruitful sources of policy for effective 
nonverbal expression in agents. Our initial study indicates that there are useful agent 
equivalents for each of Smith's ten categories of widespread vertebrate animal 
cooperation and coordination displays [60, pp. 84-105]. Some examples are discussed 
below. 

0+ : IF the current task of the PSA is of type 
uninterruptible 

THEN the PSA must blink red light until the current 
task is finished 

PRECEDENCE: A- : The PSA is forbidden from performing 
any tasks but the current one 

This policy would require the PSA to blink a red light while it is busy performing an 
uninterruptible task. During this time, it is also forbidden from performing any tasks 
but the current one. Related messages it may want to give with a similar signal might 
include: “I am unable to make contact with anybody,” “Do not attempt to 
communicate with me (for whatever reason, e.g., ‘my line is bugged').” On the 
positive side, various uses of a green light might signal messages such as: “I am open 
for calls,” “I need to talk to someone,” or “May I interject something into this 
conversation?” Displays in this general interactional category clearly have benefits for 
coordination among groups of agents by providing information about which are or are 
not in a position to interact with others, in what ways, when, and so forth. 
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0+ : IF a conversation has been initiated with someone 
THEN the PSA must face the one with whom it is 
conversing, so long as they are within the line of 
sight, until the conversation has finished 

This policy implements a kind of display associated with maintaining a previously 
established association. This display might be especially useful when the PSA is 
moving around the room and needs to let a person know that it is still attending to the 
ongoing conversation and/or task. 

0+ : IF the current task of the PSA is to move some 
distance greater than D 

THEN the PSA must signal its intention to move for S 
seconds 

PRECEDENCE: A- : The PSA is forbidden from executing its 
move 

It's no fun being hit in the head by a flying robot that suddenly decides it's got to be 
on the move. This policy prevents the PSA from moving until it has first signaled for 
some number of seconds its intention to move. Besides the pre-move signaling, some 
kind of signaling could also take place during the move itself. 

In addition to this policy regarding movement, other policies should be put in place 
to, for instance, require the PSA to stay at a safe and comfortable distance from 
people, other robotic agents, and space station structures and equipment. Of course 
the policies would take into account that different social distances may be appropriate 
in different cultures, as will be pertinent in, for example, multinational operations of 
the International Space Station. 

As our new phases of research proceed, we hope to verify the effectiveness of 
KAoS policies and services through a series of tests assessing survivability (ability to 
maintain effectiveness in the face of unforeseen software or hardware failures), safety 
(ability to prevent certain classes of dangerous actions or situations), predictability 
(assessed correlation between human judgment of predicted vs. actual behavior), 
controllability^ immediacy with which an authorized human can prevent, stop, enable, 
or initiate agent actions), effectiveness (assessed correlation between human judgment 
of desired vs. actual behavior), and adaptability (ability to respond to changes in 
context). We briefly address some aspects of adaptation next. 



6 Building Cultures for Agent Communities: 

Potential Sources of Adaptation 

There are two sorts of adaptation we believe will be critical to capture if communities 
of agents are to be enduring. The first has to do with the adaptation of policy to 
accommodate diverse contexts over which it must be applied. For example, we have 
seen an example of the need for this kind of adaptation in the last section, in which 
the comfortable distance a PSA should keep from its partner invokes cultural 
considerations. 

The limited progress we have made with regard to adaptation to context has been 
mostly in the area of adjustable autonomy and the capability it provides for functions 
like dynamic handoff of control among team members and flexible renegotiation of 
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roles and tasks among humans and agents when new opportunities arise or when 
breakdowns occur [see section 5.3 and [9]]. 

The second type of adaptation involves changes in policy, either in response to 
experience, for example, in realizing that enforcing a policy or set of policies has 
consistently resulted in untoward outcomes, or by recognizing that the nature of the 
operational world had changed in consequential ways. This second kind of adaptation 
has been even less explored. From the perspective of this paper, such adaptation 
might involve a sort of “cultural learning” that might prove challenging to current 
machine learning approaches. 

7 Conclusion 

In this paper, we have attempted to encourage an expansion of thinking about the 
sources, nature, and diversity of regulatory systems that can be utilized to achieve 
acceptable levels of order when groups of agents or mixed agent-human groups are 
engaged in consequential work. Interestingly, one impetus for this direction has been 
a desire to “make agents acceptable to humans,” for example, to make communication 
with agents natural, to make agents seem trustworthy (and actually be tmstable) in 
their participation in important affairs, and perhaps most importantly, to ensure, as in 
human societies, a kind of predictability — agents will not be acceptable to humans if 
they unexpectedly keep running amok. 

In addition, recognizing that societies need to adapt to changing conditions in 
addition to maintaining order, we have examined elements of adaptation in animal 
and human groups. Since healthy order can lapse into ineffective and unacceptable 
rigidity, we have made some brief speculations about ways elements of useful 
adaptation might coexist with those enforcing order. 

While we have focused primarily on animal signaling and order, we anticipate, 
especially in situations in which agents are embodied (e.g., physical robots), and 
move around, and act in the world, that there will be considerable benefit from 
expanding our repertoire of agent-cultural devices even farther, to include, for 
example, concrete instantiations such as “lines on the highway” or more subtle codes 
of “good manners.” 

Some years ago, Paul Wohlmuth, a philosopher of law, wrote the introductory 
chapter to an interdisciplinary special issue of the Journal of Contemporary’ Legal 
Issues focused on the “constitution of authority” [73]. Roughly interpreted, the 
constitution of authority refers to how things of various sorts come to have regulatory 
power over human conduct. 

In his introductory chapter, Wohlmuth used the simple example of an automobile 
navigating a bend in the road 22 to illustrate the ubiquity and wide variety of 
authoritative forms that come to bear on human activity. Even the basic laws of 
physics are involved. That is, there are limits to the speed with which a particular sort 
of car, on a particular sort of road, can navigate the turn without crashing, and people 
who do not want to get hurt will honor these constraints as they are able. The laws of 



22 Interestingly, Smith [60] and Nonnan [50] also draw extensively on analogies to highway 
traffic to illustrate their discussion of signaling and coordination. 
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basic physiology are in place, for instance, the eyesight, reaction time, and degree of 
alertness of the driver. Beyond these more physical constraints, all sorts of cultural 
artifacts are imparting regulatory influence. Stripes on the road demarcate the lanes 
and boundaries of the highway and whether or not the lanes may be traversed. 
Regional custom determines which side of the highway the driver should be on at all. 
Signs containing both words (e.g., “slow down”) and symbols (e.g., a twisting 
portrayal of the road section) are present. There are also inter-vehicular signalings of 
intent and disposition, and processes of coordination taking place, if there are multiple 
vehicles present. Furthermore, the appearance of a law-enforcement official, for 
example a patrolling police car, emerging on the scene, has dramatic effect on the 
behavior of the drivers. At much greater degrees of abstraction from the scene, there 
is the Motor Vehicle Code and other formal statutes that, for instance, prescribe the 
amount of certain substances that the driver may have in his or her body. In addition, 
there are entire culturally constructed deliberative bodies (e.g., the courts) 
empowered, if needed, to bridge the gap between pertinent statutes and the particulars 
of any one instance of traveling this bend. And, not much, if any, of this is static. For 
instance, if a particularly high rate of accidents results at this bend, many changes 
may take place, ranging from the physical to the more abstract. The road banking on 
the curve may be increased. The posted speed limit may be decreased. More ominous, 
scarier symbols may be posted. Consequences of violations of pertinent rules of the 
road may be made harsher. 

Societies must maintain a degree of continuity and stability; this represents a kind 
of historical memory of practices that have been effective in the past and have 
supported survival. On the other hand, the world is ever-changing; continued survival 
requires adaptations in practice to address novelty and surprise. The complexity of 
this interplay makes us realize even more that we are only at the beginning in 
addressing the dual problems of order and change in agent communities (let alone the 
optimal delicate balance between them), and it is hard not to feel a bit overwhelmed. 
But we are convinced that even little steps in understanding how to better incorporate 
the content and mechanisms of culture into agent societies will be both interesting and 
fruitful. 



Acknowledgements 

The authors gratefully acknowledge the sponsorship of this research by the NASA 
Cross-Enterprise and Intelligent Systems Programs, and a joint NASA-DARPA IT AC 
grant. Additional support was provided by DARPA's CoABS, Ultra*Log, EPCA, and 
Augmented Cognition programs, by the Army Research Lab's Advanced Decision 
Architectures program (ARLADA), and the Office of Naval Research. A special 
thanks is due to the following individuals for their valuable contributions to the ideas 
developed in this paper: Alessandro Acquisti, James Allen, Guy Boy, Kathleen 
Bradshaw, Maggie Breedy, Larry Bunch, Murray Burke, Nate Chambers, Bill 
Clancey, Greg Dorais, Joan Feltovich, Ken Ford, Lucian Galescu, Mark Greaves, Jack 
Hansen, Pat Hayes, Robert Hoffman, Matt Johnson, Hyuckchul, Jung, Shri Kulkarni, 
Henry Lieberman, Cheryl Martin, Cindy Mason, Robin Murphy, Nicola Muscettola, 
Don Norman, Jerry Pratt, Anil Raj, Dylan Schmorrow, Debra Schreckenghost, Mike 




Social Order and Adaptability in Animal and Human Cultures 45 



Shafto, Maarten Sierhuis, Austin Tate, Milind Tambe, William Taysom, Ron Van 
Hoof, the late Paul Wohlmuth, and Tim Wright. 



References 

[1] Acquisti, A., Sierhuis, M., Clancey, W. J., Bradshaw, J. M. (2002). Agent-based 
modeling of collaboration and work practices onboard the International Space Station. 
Proceedings of the Eleventh Conference on Computer-Generated Forces and Behavior 
Representation. Orlando, FL, USA. 

[2] Allen, J. F., Ferguson, G. (2002). Human-machine collaborative planning. Proceedings of the 
NASA Planning and Scheduling Workshop. Houston, TX, USA. 

[3] Asimov, I. (1942/1968). Runaround. In I. Asimov (Ed.), I, Robot, (pp. 33-51). London, 
England: Grafton Books. Originally published in Astounding Science Fiction, 1942, pp. 94- 
103. 

[4] Barber, K. S., Gamba, M., Martin, C. E. (2002). Representing and analyzing adaptive 
decision-making frameworks. In H. Hexmoor, C. Castelfranchi, R. Falcone (Ed.), Agent 
Autonomy, (pp. 23-42). Dordrecht, The Netherlands: Kluwer. 

[5] Boella, G. (2002). Obligations and cooperation: Two sides of social rationality. In H. 
Hexmoor, C. Castelfranchi, R. Falcone (Ed.), Agent Autonomy, (pp. 57-78). Dordrecht, The 
Netherlands: Kluwer. 

[6] Boman, M. (1999). Norms in artificial decision-making. Artificial Intelligence and Law, 7, 
17-35. 

[7] Bradshaw, J. M., Beautement, P., Raj, A., Johnson, M., Kulkami, S., Suri, N. (2003). Making 
agents acceptable to people. In N. Zhong J. Liu (Ed.), Intelligent Technologies for 
Information Analysis: Advances in Agents, Data Mining, and Statistical Learning, (pp. in 
press). Berlin: Springer Verlag. 

[8] Bradshaw, J. M„ Boy, G., Durfee, E., Gruninger, M„ Hexmoor, H„ Suri, N., Tambe, M., 
Uschold, M., Vitek, J. (Ed.). (2003). Software Agents for the Warfighter. ITAC Consortium 
Report. Cambridge, MA: AAAI Press/The MIT Press. 

[9] Bradshaw, J. M., Jung, H., Kulkami, S., Taysom, W. (2004). Dimensions of adjustable 
autonomy and mixed-initiative interaction. In M. Klusch, G. Weiss, M. Rovatsos (Ed.), 
Computational Autonomy, (pp. in press). Berlin, Germany: Springer-Verlag. 

[10] Bradshaw, J. M., Sierhuis, M., Acquisti, A., Feltovich, P., Hoffman, R., Jeffers, R., Prescott, 
D., Suri, N., Uszok, A., Van Hoof, R. (2003). Adjustable autonomy and human-agent 
teamwork in practice: An interim report on space applications. In H. Hexmoor, R. Falcone, C. 
Castelfranchi (Ed Agent Autonomy, (pp. 243-280). Kluwer. 

[11] Bradshaw, J. M., Sierhuis, M., Gawdiak, Y., Jeffers, R., Suri, N., Greaves, M. (2001). 
Adjustable autonomy and teamwork for the Personal Satellite Assistant. Proceedings of the 
IJCAI-01 Workshop on Autonomy, Delegation, and Control: Interacting with Autonomous 
Agents. Seattle, WA, USA. 

[12] Bradshaw, J. M., Suri, N„ Breedy, M. R., Canas, A., Davis, R., Ford, K. M„ Hoffman, R., 
Jeffers, R., Kulkami, S., Lott, J., Reichherzer, T„ Uszok, A. (2002). Terraforming 
cyberspace. In D. C. Marinescu C. Lee (Ed.), Process Coordination and Ubiquitous 
Computing, (pp. 165-185). Boca Raton, FL: CRC Press. Updated and expanded version of an 
article that originally appeared in IEEE Intelligent Systems, July 2001, pp. 49-56. 

[13] Breazeal, C., Scassellati, B. (1999). How to build robots that make friends and influence 
people. IROS. Kyonjiu, Korea. 

[14] Calmet, J., Daemi, A., Endsuleit, R., Mie, T. (2004). A liberal approach to openess in 
societies of agents. In this volume. 

[15] Clark, H. H. ( 1 996). Using Language. Cambridge, UK: Cambridge University Press. 

[16] Clarke, R. (1993-1994). Asimov's laws of robotics: Implications for information technology, 
Parts 1 and 2. IEEE Computer \ December/January, 53-61/57-66. 




46 Paul J. Feltovich et al. 



[17] Cohen, P. R., Levesque, H. J. (1991). Teamwork. Technote 504. Menlo Park, CA: SRI 
International, March. 

[18] Cohen, R., Fleming, M. (2002). Adjusting the autonomy in mixed-initiative systems by 
reasoning about interaction. In H. Hexmoor, C. Castelfranchi, R. Falcone (Ed.), Agent 
Autonomy, (pp. 105-122). Dordrecht, The Netherlands: Kluwer. 

[19] Conte, R., Castelfranchi, C. (1995). Cognitive and social action. London, England: UCL 
Press. 

[20] Cook, R. I., Woods, D. D. (1994). Operating at the sharp end: The complexity of human 
error. In S. U. Bogner (Ed.), Human Error in Medicine. Hillsdale, NJ: Lawrence Erlbaum. 

[21] d'Invemo, M., Luck, M. (2001). Understanding Agent Systems. Berlin, Germany: Springer- 
Verlag. 

[22] Damianou, N., Dulay, N., Lupu, E. C., Sloman, M. S. (2000). Ponder: A Language for 
Specifying Security and Management Policies for Distributed Systems, Version 2.3. Imperial 
College of Science, Technology and Medicine, Department of Computing, 20 October 2000. 

[23] Dorais, G„ Bonasso, R. P., Kortenkamp, D., Pell, B., Schrekenghost, D. (1999). Adjustable 
autonomy for human-centered autonomous systems on Mars. Proceedings of the AAAI Spring 
Symposium on Agents with Adjustable Autonomy. AAAI Technical Report SS-99-06. Menlo 
Park, CA, Menlo Park, CA: AAAI Press. 

[24] Dorais, G., Desiano, S. D., Gawdiak, Y., Nicewarmer, K. (2003). An autonomous control 
system for an intra-vehicular spacecraft mobile monitor prototype. Proceedings of the 
Seventh International Symposium on Artificial Intelligence, Robotics, and Automation in 
Space (i-SAIRAS 2003). Nara, Japan. 

[25] Falcone, R., Castelfranchi, C. (2002). From automaticity to autonomy: The frontier of 
artificial agents. In H. Hexmoor, C. Castelfranchi, R. Falcone (Ed.), Agent Autonomy, (pp. 79- 
103). Dordrecht, The Netherlands: Kluwer. 

[26] Feltovich, P., Bradshaw, J. M., Jeffers, R., Uszok, A. (2003). Order and KAoS: Using policy 
to represent agent cultures. Proceedings of the AAMAS 03 Workshop on Humans and Multi- 
Agent Systems. Melbourne, Australia. 

[27] Fox, J., Das, S. (2000). Safe and Sound: Artificial Intelligence in Hazardous Applications. 
Menlo Park, CA: AAAI Press/The MIT Press. 

[28] Gawdiak, Y., Bradshaw, J. M., Williams, B„ Thomas, H. (2000). R2D2 in a softball: The 
Personal Satellite Assistant. H. Lieberman (Ed.), Proceedings of the ACM Conference on 
Intelligent User Interfaces (IUI 2000), (pp. 125-128). New Orleans, LA, New York: ACM 
Press, 

[29] Geertz, C. (1973). The Interpretation of Cultures. New York, NY: Basic Books. 

[30] Greaves, M„ Holmback, H., Bradshaw, J. M. (1999). What is a conversation policy? M. 
Greaves, J. M. Bradshaw (Ed.), Proceedings of the Autonomous Agents '99 Workshop on 
Specifying and Implementing Conversation Policies, (pp. 1-9). Seattle, WA, USA. 

[31] Hancock, P. A., Scallen, S. F. (1998). Allocating functions in human-machine systems. In R. 
Hoffman, M. F. Sherrick, J. S. Warm (Ed.), Viewing Psychology as a Whole, (pp. 509-540). 
Washington, D.C.: American Psychological Association. 

[32] Hohfeld, W. N. (1913). Fundamental legal conceptions as applied injudicial reasoning. Yale 
Law Journal, 23. 

[33] Horvitz, E. (1999). Principles of mixed-initiative user interfaces. Proceedings of the ACM 
SIGCHI Conference on Human Factors in Computing Systems (CHI '99). Pittsburgh, PA, 
New York: ACM Press. 

[34] Horvitz, E., Jacobs, A., Hovel, D. (1999). Attention-sensitive alerting. Proceedings of the 
Conference on Uncertainty and Artificial Intelligence (UAI '99), (pp. 305-313). Stockholm, 
Sweden. 

[35] Johnson, M., Chang, P., Jeffers, R., Bradshaw, J. M., Soo, V.-W., Breedy, M. R., Bunch, L., 
Kulkami, S., Lott, J., Suri, N., Uszok, A. (2003). KAoS semantic policy and domain services: 
An application of DAML to Web services-based grid architectures. Proceedings of the 
AAMAS 03 Workshop on Web Services and Agent-Based Engineering. Melbourne, Australia. 

[36] Jordan, M., Atkinson, M. (1998). Orthogonal persistence for Java — A mid-term report. Sun 
Microsystems Laboratories. 




Social Order and Adaptability in Animal and Human Cultures 47 



[37] Kagal, L., Finin, T., Joshi, A. (2003). A policy language for pervasive systems. Proceedings 
of the Fourth IEEE International Workshop on Policies for Distributed Systems and 
Networks, (pp. http://umbc.edu/~finin/papers/policy03.pdf). Lago di Como, Italy. 

[38] Kahn, M., Cicalese, C. (2001). CoABS Grid Scalability Experiments. O. F. Rana (Ed.), 
Second International Workshop on Infrastructure for Scalable Multi-Agent Systems at the 
Fifth International Conference on Autonomous Agents. Montreal, CA, New York: ACM 
Press. 

[39] Knoll, G., Suri, N., Bradshaw, J. M. (2001). Path-based security for mobile agents. 
Proceedings of the First International Workshop onthe Security of Mobile Multi-Agent 
Systems (SEMAS-2001) at the Fifth International Conference on Autonomous Agents (Agents 
2001), (pp. 54-60). Montreal, CA, New York: ACM Press. 

[40] Krogh, C., Herrestad, H. (1999). Hohfeld in Cyberspace and other applications of normative 
reasoning in agent technology. Artificial Intelligence and Law, 7(1), 81-96. 

[41] Lopez y Lopez, F., Luck, M., d'Invemo, M. (2001). A framework for norm-based inter-agent 
dependence. Proceedings of the Third Mexican Internation Conference on Computer Science. 

[42] Lopez y Lopez, F., Luck, M., d'Invemo, M. (2002). Constraining autonomy through norms. 
Proceedings of the Conference on Autonomous Agents and Multi-Agent Systems, (pp. 674- 
681). Bologna, Italy. 

[43] Maes, P. (1997). Agents that reduce work and information overload. In J. M. Bradshaw (Ed.), 
Software Agents, (pp. 145-164). Cambridge, MA: AAAI Press/The MIT Press. 

[44] Mayr, E. (1997). This Is Biology. Cambridge, MA: Belkamp Press of Harvard University 
Press. 

[45] McBumey, P., Parsons, S. (2004). Engineering democracy in open agent systems. In this 
volume. 

[46] Morris, D. (2002). Peoplewatching. London, England: Vintage. 

[47] Murphy, R. R. (2000). Introduction to AI Robotics. Cambridge, MA: The MIT Press. 

[48] Myers, K., Morley, D. (2003). Directing agents. In H. Hexmoor, C. Castelfranchi, R. Falcone 
(Ed.), Agent Autonomy, (pp. 143-162). Dordrecht, The Netherlands: Kluwer. 

[49] Neville, B., Pitt, J. (2004). A computational framework for social agents in agent-mediated e- 
commerce. In this volume. 

[50] Norman, D. A. (1992). Turn signals are the facial expressions of automobiles. In Turn Signals 
Are the Facial Expressions of Automobiles, (pp. 117-134). Reading, MA: Addison-Wesley. 

[51] Norman, D. A. (1997). How might people interact with agents? In J. M. Bradshaw (Ed.), 
Software Agents, (pp. 49-55). Cambridge, MA: The AAAI Press/The MIT Press. 

[52] Pynadath, D., Tambe, M. (2001). Revisiting Asimov's first law: A response to the call to 
arms. Proceedings of ATAL 01. 

[53] Sawyer, R. K. (2001). Creating Conversations: Improvisation in Everyday Discourse. 
Cresskill, NJ: Hampton Press. 

[54] Scerri, P., Pynadath, D., Tambe, M. (2002). Adjustable autonomy for the real world. In H. 
Hexmoor, C. Castelfranchi, R. Falcone (Ed .), Agent Autonomy, (pp. 163-190). Dordrecht, The 
Netherlands: Kluwer. 

[55] Schreckenghost, D., Martin, C., Bonasso, P., Kortenkamp, D., Milam, T., Thronesbery, C. 
(2003). Supporting group interaction among humans and autonomous agents. Submitted for 
publication. 

[56] Schreckenghost, D., Martin, C., Thronesbery, C. (2003). Specifying organizational policies 
and individual preferences for human-software interaction. Submitted for publication. 

[57] Sergot, M. (2001). A computational theory of normative positions. ACM Transactions on 
Computational Logic, 2(4), 581-622. 

[58] Shoham, Y., Tennenholtz, M. (1992). On the synthesis of useful social laws for artificial 
agent societies. Proceedings of the Tenth National Conference on Artificial Intelligence, (pp. 
276-281). San Jose, CA, USA'. 

[59] Sierhuis, M., Bradshaw, J. M., Acquisti, A., Van Hoof, R., Jeffers, R., Uszok, A. (2003). 
Human-agent teamwork and adjustable autonomy in practice. Proceedings of the Seventh 
International Symposium on Artificial Intelligence, Robotics and Automation in Space (i- 
SAIRAS). Nara, Japan. 




48 Paul J. Feltovich et al. 



[60] Smith, W. J. (1977). The Behavior of Communicating. Cambridge, MA: Harvard University 
Press. 

[61] Smith, W. J. (1995). The biological bases of social attunement. Journal of Contemporary 
Legal Issues, 6. 

[62] Suri, N„ Bradshaw, J. M., Breedy, M. R., Groth, P. T„ Hill, G. A., Jeffers, R. (2000). Strong 
Mobility and Fine-Grained Resource Control in NOMADS. Proceedings of the 2nd 
International Symposium on Agents Systems and Applications and the 4th International 
Symposium on Mobile Agents (ASA/MA 2000). Zurich, Switzerland, Berlin: Springer- Verlag. 

[63] Suri, N., Bradshaw, J. M., Breedy, M. R., Groth, P. T., Hill, G. A., Jeffers, R., Mitrovich, T. 
R., Pouliot, B. R., Smith, D. S. (2000). NOMADS: Toward an environment for strong and 
safe agent mobility. Proceedings of Autonomous Agents 2000. Barcelona, Spain, New York: 
ACM Press. 

[64] Suri, N., Bradshaw, J. M., Burstein, M. H„ Uszok, A., Benyo, B., Breedy, M. R„ Carvalho, 
M„ Diller, D„ Groth, P. T„ Jeffers, R„ Johnson, M., Kulkarni, S„ Lott, J. (2003). DAML- 
based policy enforcement for semantic data transformation and filtering in multi-agent 
systems. Proceedings of the Autonomous Agents and Multi-Agent Systems Conference 
(AAMAS 2003). Melbourne, Australia, New York, NY : ACM Press. 

[65] Suri, N., Carvalho, M., Bradshaw, J. M„ Breedy, M. R., Cowin, T. B., Groth, P. T., 
Saavendra, R., Uszok, A. (2003). Mobile code for policy enforcement. Policy 2003. Como, 
Italy. 

[66] Tarnbe, M., Shen, W., Mataric, M., Pynadath, D. V., Goldberg, D., Modi, P. J., Qiu, Z., 
Salemi, B. (1999). Teamwork in cyberspace: Using TEAMCORE to make agents team-ready. 
Proceedings of the AAAI Spring Symposium on Agents in Cyberspace. Menlo Park, CA, 
Menlo Park, CA: The AAAI Press. 

[67] Tonti, G., Bradshaw, J. M., Jeffers, R., Montanari, R., Suri, N., Uszok, A. (2003). Semantic 
Web languages for policy representation and reasoning: A comparison of KAoS, Rei, and 
Ponder. In D. Fensel, K. Sycara, J. Mylopoulos (Ed.), The Semantic Web — ISWC 2003. 
Proceedings of the Second International Semantic Web Conference, Sanibel Island, Florida, 
USA, October 2003, LNCS 2870. (pp. 419-437). Berlin: Springer. 

[68] Trevarthen, C. (1979). Communication and cooperation in early infancy: A description of 
primary intersubjectivity. In M. Bullowa (Ed.), Before Speech, (pp. 321-348). Cambridge, 
England: Cambridge University Press. 

[69] Uszok, A., Bradshaw, J. M., Hayes, P., Jeffers, R., Johnson, M„ Kulkarni, S., Breedy, M. R., 
Lott, J., Bunch, L. (2003). DAML reality check: A case study of KAoS domain and policy 
services. Submitted to the International Semantic Web Conference (ISWC 03). Sanibel Island, 
FL, USA. 

[70] Vanderbilt, A. (1952/1963). Amy Vanderbilt's New Complete Book of Etiquette: The Guide to 
Gracious Living. Garden City, NY : Doubleday and Company. 

[71] Verhagen, H. (2001). Norms and artificial agents. Sixth Meeting of the Special Interest 
Group on Agent-Based Social Simulation, ESPRIT Network of Excellence on Agent- 
Based Computing. Amsterdam, Holland, http://abss.cfpm.org/amsterdam- 
01/abssnorms.pdf. 

[72] Weld, D„ Etzioni, O. (1994). The firsts law of robotics: A call to arms. Proceedings of 
the National Conference on Artificial Intelligence (AAAI 94), (pp. 1042-1047). 

[73] Wohlmuth, P. C. (1995). Traveling the highway: Sources of momentum in behavioral 
regulation. Journal of Contemporary Legal Issues, 6, 1-9. 




Using Swarm Intelligence in Linda Systems* 



Robert Tolksdorf 1 and Ronaldo Menezes 2 

1 Freie Universitat Berlin 
Institut fiir Informatik 
AG Netzbasierte Informationssysteme 
Takustr. 9, D- 14195 Berlin, Germany 
research@robert-tolksdorf . de 
2 Florida Institute of Technology 
Department of Computer Sciences 
150 W. University Blvd., Melbourne, FL 32901, USA 
rmenezesOcs . f it . edu 



Abstract. Natural forming multi-agent systems can grow to enormous 
sizes and perform seemingly complex tasks without the existence of any 
centralized control. Their success comes from the fact that agents are 
simple and the interaction with the environment and neighboring agents 
is local in nature. We describe how swarm intelligence can be used in 
the implementation of a Linda-based system called SwarmLinda. We 
argue that SwarmLinda achieves many desired characteristics such as 
scalability, adaptiveness and even some level of fault-tolerance. 



1 Introduction 

In the past 20 years, coordination models, and in particular Linda, have proven 
to be quite successful in tackling the intricacies of medium-to-large-scale open 
systems. Yet, Linda systems may not scale well with the number of tuple spaces 
and processes. One of the reasons for the poor scalability is that the design 
of these systems still inherits ideas from early Linda systems [1, 2] which were 
focused on parallel computing. 

In the quest for identifying systems that are scalable we turned away from 
standard techniques and looked in the field of Biology, more specifically Zoology 
where one can find several examples of scalable natural forming multi-agents 
systems (aka Swarms) . Swarms are notorious for their organization despite their 
enormous sizes. Their activities are based on simple rules that can be easily 
implemented as computer programs. Their interaction involves only local com- 
munications. 

We focus on the use of swarm-based techniques (such as approaches based 
on ant-colonies [3]) in tuple space systems. Our goal is to tackle the scalability 
problem studying how to improve the current scenario using techniques adapted 
from models originating from biological collective organisms. 

* This work has been partially funded by the DAAD (Germany) and the NSF (USA) 
contract numbers D/03/34491 and INT-0337161 respectively. 
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Fig. 1 . Centralized tuple spaces 



1.1 Scalability of Linda Systems 

Scalability of tuple space systems is a well studied topic. Studies range from 
theoretical works [4] to implementations of tuple-space systems that claim to 
be scalable [5, 6], to studies on the suitability of these systems to wide-area 
computing [7]. 

Theoretical works tend to formalize the concept of tuple spaces and use the 
formalism as a way to show how tuples should be distributed. For instance, Obre- 
iter and Graf [4] argue that scalability can be achieved by organizing tuples in 
servers based on the tuples’ structure. In the practical arena, Rowstron did sev- 
eral works attempting to make Linda systems more scalable [8, 6]. His approach 
is based on the idea of configuring tuple spaces hierarchically and classifying 
them into local and global tuple spaces. 

As an illustration of the communication overhead in existing Linda imple- 
mentations, consider the problem of retrieving a tuple (based on a given tem- 
plate) from a set of remote servers. A common approach is to ask a set of servers 
for matching tuples by some multicast. Each server searches, matches and locks 
the candidate tuples while it offers them to the requestor. This scheme estab- 
lishes a state “tuple under request” which is global to the set of servers and the 
requestor. 

The above approaches assume that improved scalability depends solely on 
the way the data is distributed. To be of use to Linda systems, these data- 
oriented techniques need to be augmented with other concepts to minimize the 
communication overhead. 



1.2 Distributing Linda 

The literature on describes a plethora of approaches for distributing tuple spaces: 

— Centralization is a simple client-server distribution strategy where one spe- 
cific server-machine operates the complete tuple space (as in TSpaces [9]). 
The tuple space can be accessed by clients that are arbitrarily located on 
a network (see Fig. 1). 

The centralized tuple-space server has the advantage of an easy imple- 
mentation that basically attaches a network interface to an otherwise non- 
distributed tuple-space. However, it carries all the disadvantages of any cen- 
tralization of services. 

— Partitioning of tuple spaces is the strategy to co-locate tuples with common 
characteristics in one of a set of tuple-space servers; requests with the same 
characteristics are then routed towards that node (see Fig. 2). While simple 
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Fig. 2. Partitioned tuple spaces 




partitioning might lead to unbalanced partitions, a carefully chosen hashing 
function on tuples can do better [10]. 

Partitioning provides a distributed management of tuple spaces including 
concurrent execution of operations on the partitions. But it does include cen- 
tralization for certain sets of tuple and handles reconfigurations very poorly. 

— Full replication places complete copies of tuple spaces on several nodes at 
different locations. Any addition and removal of data has to be replicated 
to all nodes so that searches for data can be performed locally (see Fig. 3). 
It distributes the load for data-searches and offers some support for fault- 
tolerance. However, the cost of keeping the replicas consistent is high and it 
requires locking protocols or equivalents [11]. 

— Intermediate replication is a schema that uses a grid of nodes formed by 
logical intersecting “busses” [12]. Each node is part of exactly one outbus and 
one inbus. Data is replicated on all nodes of the outbus, whereas searches are 
performed on the inbus. As inbusses intersects all outbusses, they provides 
a complete view of the tuple space (see Fig. 4). 

This scheme allows for as many concurrent replications and searches for 
data as there are out- and inbusses respectively. The removal of some data, 
however, requires a consistent update on all nodes on the respective outbus. 

None of these approaches are long-term solutions. Our proposal is SwarmLinda. 



2 Concepts of a SwarmLinda 

Over the past few years, new models originating from biology have been studied 
in the field of computer science [13, 14, 15, 16]. The primary interest lies on using 
these models as techniques for finding feasible solutions to NP-hard problems. 



52 



Robert Tolksdorf and Ronaldo Menezes 




Fig. 4. Intermediate replication 



In these models, actors (ants, termite, etc.) sacrifice individual goals (if any) 
for the benefit of the collective. They act extremely decentralized. The work is 
carried out by making purely local decisions and by taking actions that require 
only very small computations. These characteristics alone enable these systems 
to scale very well. This swarm intelligence is an interesting opportunity to rethink 
scalability of tuple spaces. 

The use of such principles as an alternative to data-oriented schemes could 
simply consist of a system where templates are modeled as ants that search for 
food (the matching tuples) . One can understand the “world” of Linda servers as 
a two-dimensional space in which ants search for food, leaving trails to successful 
matches. 

With an ant-based optimization of the trails, shortest tuple-producer/- 
consumer paths can be found and used to optimize system performance: instead 
of querying sets of replicas, the “template-ant” goes directly to where it expects 
a match. Technically, this accounts to a single message interchange between the 
producing and consuming sites. 

The above is just an illustration. A more realistic SwarmLinda should con- 
sider a few principles that can be observed in most swarm systems [3] : 

Simplicity: Swarm individuals are simple creatures that perform simple tasks. 
They do no deep reasoning and implement a small and set of simple rules. 
The execution of these rules leads to the emergence of complex behavior. 
Active entities in a SwarmLinda should also obey the principle of simplicity 
and be “small” in terms of resource usage. 

Dynamism: Natural swarms adapt to dynamically changing environments. In 
open distributed systems, the configuration of running applications and ser- 
vices changes over time. If a tuple is found in a given location it does not 
necessarily mean that other similar tuples will exist in the same location in 
the future. 

Locality: Swarm individuals observe their direct neighborhood and take deci- 
sions based on their local view. As the key to scalability in SwarmLinda, 
active entities have to perform only local searches and inquire only to direct 
neighbors. 
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To fully appreciate these principles, one needs to understand how a Swarm- 
Linda should be organized. Linda systems do not have the idea of ants or food. 
A SwarmLinda needs to abstract these concepts in the Linda world. 

We will base the description of a SwarmLinda on the following abstractions. 
Individuals are the active entities that are able to observe their neighborhood, to 
move in the environment, and to change the state of the environment in which 
they are located. The environment is the context in which the individuals work 
and observe. The state is an aspect of the environment that can be observed and 
changed by individuals. 

3 Algorithms for a SwarmLinda 

We can now focus on bringing the concepts above to life by defining the concrete 
environment and its state, and the individuals and the rules they apply (based 
on [14, 3]). The description here is succinct. For a full description refer to [17]. 

3.1 Searching for Tuples 

Ants look for food in the proximity of the ant hill. Once found, the food is 
brought to the ant hill and a trail is left so that other ants can know where food 
is. The ants know the way back to the ant hill because they have a short memory 
of the last few steps they took and also because the ant hill has a distinctive scent 
that can be tracked by the ants. One could view tuples as food, the locations 
where the tuples are stored can be seen as the terrain while the templates are 
seen as ants that wander in the locations in search of tuples. The ant hill is the 
process that executed the operation. 

The individuals are the template-ants, the environment consists of tuple- 
space nodes whose state is composed by the tuples stored and “scent” of different 
kinds of templates indicating the likelihood that matches for that template are 
available. The scents are volatile and disappear slowly. The tuple-searching ants 
follow the properties below: 

1. The first step is to spread the scent of the process in the node it is connected 
and that node’s neighborhood to represent the ant hill. 

2. Check for a matching tuple at the ant’s current location. If a match is found, 
return to the origin location and leave scent for the template matched at each 
step. If no match is found, check the direct neighborhood for traces of the 
desired scent. 

3. If there are no desired scents in the direct neighborhood of the ant’s lo- 
cation around the current location, randomly choose a direction from the 
neighboring nodes to look for a tuple. 

4. If there is a scent that indicates a direction for next step (matching 
scent), move one step towards that scent and start over. We achieve non- 
determinism by adding a small random factor in a range of [—£,£] to each 
scent. This enables paths other that the one with the strongest scent to be 
chosen. 
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5. After each unsuccessful step without a match, the ant stops its search with 
a probability 7 . 7 is initially 0 and it is increased by some r with each 
unsuccessful step, T itself also increases over time. When the ant decides to 
stop searching, it takes one of three actions: 

(a) Sleep for some time and then continue. 

(b) Die and be reborn after some time at the location the search started. 

(c) Materialize in some other random location and continue to search for 
tuples. 

Which action to take depends on the age of the ant. After it has slept several 
times, it tries a rebirth. After some re-births, it decides to rematerialize 
elsewhere. 

The result of the above is the emergence of application specific paths between 
tuple producers and consumers. Given that scents are volatile, the paths found 
can dynamically adapt to changes in the system - when consumers or producers 
join, leave or move within the system. 

An additional concept needs to be mentioned: ants do not tend to change 
their direction radically - given no influence of scents. If an ant comes from one 
direction, the opposite direction has the highest probability to be chosen. 

Compare this searching approach with a hashing mechanism. Tuples are nor- 
mally searched based on a hash function that takes the template as the input and 
generate a location where the tuple can be found. Hashing is definitely fast but 
unfortunately not very adaptive. The determinism that exist in hash functions 
forces tuples with the same template to always be placed in the same location 
no matter the size of the system, thus causing bottleneck problems if tuples 
matching such template are highly demanded. 

In SwarmLinda, tuples matching the same template rather tend to stay to- 
gether. However, if such tuples are being produced in locations that are far 
enough from each other the tuples will remain separated and clusters across the 
system will be created. This should avoid the creation of bottlenecks when tu- 
ples of a certain template are required by several processes. As searches will be 
started from various locations, tuples should be retrieved from the closest cluster 
of tuples. 

Another problem with hashing approaches is that they are not fault tolerant. 
When using swarming, failures are just another change in the environment - they 
behave like ants trying to search for food in a food supply that was suddenly 
destroyed. 

3.2 Distribution Mechanism 

Another area of SwarmLinda where swarming abstractions may be used is in 
the distribution of tuples amongst the nodes. Historically, tuples have been dis- 
tributed using various static mechanisms (see Section 1.2). In SwarmLinda the 
partitioning of tuple spaces can be dynamic using the ants concept of brood 
sorting. 
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Ants are able to sort different kinds of things kept in the ant hill such as 
food, larvae, eggs, etc. In an ant hill these are normally sorted by their type. 
More importantly, ants do this process in spite of the amount of each type, thus 
being very scalable. The individuals that operate here are tuple-ants (as opposed 
to template-ants). The environment is unchanged - it remains as a network of 
nodes. The state is the set of tuples stored. 

A SwarmLinda implementation may use brood sorting for tuple distribution. 
In this process, tuples are the food and the ant is the active process representing 
an out: 

1. Upon the execution of an out primitive, start visiting the nodes. 

2. Observe the kind of tuples (the template they match) the nodes store. Each 
out-ant should have a limited memory to base that decision on local infor- 
mation. 

3. Store the tuple in the node if nearby nodes store tuples matching the same 
template. Again this decision also considers a small random factor [—£,£]. 

4. If nearby nodes do not contain similar tuples, randomly choose (using the 
random factor) whether to drop or continue to carry the tuple to another 
node. 

To guarantee that the steps above work well, certain conditions must be satisfied. 
The out-ant should be able to store the tuple eventually. For each time the 
process decides not to store the tuple, the random factor will tend to £. This 
increases the chance of storing the tuple in the next step. Also the likelihood 
to store the tuple is calculated stochastically based on the kinds of objects in 
memory - if most of the objects in memory are similar to the one being carried 
out the likelihood to store the tuple becomes very high. 

The power of this approach shows when it is compared with the partition- 
ing scheme from Section 1.2 since it can improve the availability of the system 
without counting on costly techniques such data replication. In the ant-based ap- 
proach, there are no assumptions about the behavior of applications, there are no 
pre-defined distribution schema, and there are no special scenarios implemented 
to deal with failures in the node. 



3.3 Dealing with Openness 

Openness is known to be one of the main challenges in distributed systems - the 
ability of a system to deal with changes can be a great asset. For SwarmLinda to 
show adaptive behavior for collections of similar tuples, we again use tuple-ants 
as the individuals and the environment as the terrain of nodes that has scents 
as the state. 

For a SwarmLinda, we want tuples matching the same template to be kept 
together (as described in Section 3.2) but we do not want them to be fixed to 
a given location. Consider a function Sc : T —y S on templates and tuples and 
a relation C : S x S on scent that defines similarity of scent. When template te 
and tuple tu match, then Sc(te), Sc(tu) £ C. 



56 



Robert Tolksdorf and Ronaldo Menezes 



1. A new tuple-ant that carries a tuple tu emits Sc{tu ) at its origin. A new 
template-ant that carries a template te emits Sc(te ) at its origin. 

2. Template-ants remain at that position and never move. 

3. Tuple-ants sense their environment for a scent similar to Sc(tu) - as given 
by C. If there is such scent, then other template- or tuple-ants are around. 

4. Based on the strength of the detected scent plus the small random factor 
[—£,£], the tuple-ant decides to move into that direction or to stay where it 
is. 

The above properties causes tuples to stay closer to where other similar tuples 
are needed or are being produced even if this consists of migrating from one 
node to another. This would affect on the distribution mechanism explained in 
Section 3.2. When a tuple is being stored, the scent left by in and out primitives 
should also be considered when deciding to drop the tuple in the current node 
or to keep “walking” through the nodes searching for a good place to drop the 
tuple. 



3.4 Balancing Tuple- and Template Movement 

So far, we always identified either the tuple-ants or the template-ants as individ- 
uals that move and perform a continued search. Here we describe an interme- 
diate approach where ants can be both tuples and templates. Every tuple- and 
template-ant decides after its birth whether it goes out to other nodes seeking 
matches or stays at its origin until it is found by some other ant. 

Consider an application where a node consumes a lot of tuples that are gen- 
erated on other nodes. If trails from the producers to the consumer are found, it 
makes no sense to have the consumer start template-ants that seek the produc- 
ers. Based on the system history (of scents), it is known where a consumer is and 
what the path is, so tuple-ants should be started there while the template-ants 
at the consumer should remain stationary and wait for a matching tuple-ant. 
But if the former consumer starts to be a producer after the calculation of some 
results, it becomes reasonable to start template-ants from the former producers 
to search for the result. 

For the algorithm, the individuals are tuple- and template-ants. The environ- 
ment is the terrain of nodes. Their states include two scents: The visitor scent 
indicates whether the location is visited successfully by other ants (it is an at- 
traction) or not (it is an outsider). Success means that the visiting ant found 
matches at this location. The producer- consumer scent ranges over [—</), (f>\. Pos- 
itive values indicate that the matches that took place were such that a visiting 
template-ant retrieved a tuple from that location - showing that the location 
is a producer of information. A negative scent indicates that visiting tuple-ants 
were matched with a template - the location is a consumer of tuples. 

Tuple- and template-ants follow the algorithms from Section 3.1 to find 
matching templates resp. tuples. If a tuple-ant finds a match, it neutralizes a bit 
of producer-consumer scent at the location. When a template-ant finds a match, 



Using Swarm Intelligence in Linda Systems 



57 



it adds a bit of this scent at the location. Both kinds of ants leave a bit of visitor 
scent in the case of success. 

New ants are either tuple- or a template-ants depending on the kind of oper- 
ation requested. A new tuple-ant emits a bit of producer-consumer scent at the 
location of its birth, a template-ant neutralizes some. These ants can behave in 
two different ways: Either they are active and move around following the search 
algorithms as described above, or they are passive and remain at their origin to 
be found by others. 

The further fate of a new ant depends on the current state of their location 
based on the visitor- and producer-consumer-scent: 



Producer 


Consumer 


Attraction Passive tuple-ant 


Passive/active template-ant 


Active/passive tuple-ant 


Passive template-ant 


Outsider Active tuple-ant 


Passive template-ant 


Passive tuple-ant 


Active template-ant 



If a producer is visited by many ants, there is no need to send out tuple-ants. 
Template-ants can be passive too as many visitors satisfy them, but also active 
to establish a global balance. The ratio passive/active could be adjusted. For 
an attractive consumer, template-ants can remain passive. Tuple-ants might be 
active with the same argument on a global balance. If a producer is only rarely 
visited, it will send out its tuples-ants to find matches. Its template-ants can 
remain passive as they are not many. 

If a consumer is not visited by many other ants, it will send out active 
template-ants to find matches and generate passive tuple-ants to attract other 
active template-ants to improve the chance of becoming an attraction. 

3.5 Analysis 

The algorithms described in this demonstrate the power of swarm abstractions in 
the context of Linda systems. In particular, SwarmLinda appears to be efficient 
dealing with (at least) the following issues: 

Scalability: The decisions taken by ants are based on local observations like 
sensing the direct environment or asking the currently visited node for 
a match. State changes are also purely local like changing the scent at the 
current or directly surrounding locations. There are no global states like 
locking at a set of nodes. With that, the activities of the algorithms are 
independent of the number of active ants. 

Adaptiveness: The decisions taken by ants are based on the current state of the 
system. This state is changed based on decisions of individuals in the system 
by leaving fresh scent at locations. The influence of prior decisions becomes 
smaller over time with the evaporation of old scent. The state therefore reflect 
the current configuration of the system in terms of where and what kind of 
tuples are produced and consumed. The behavior of the ants dynamically 
adapts to this configurations. 
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Fault-Tolerance: The influence of wrong decisions of ants are only temporary. 
Changes in scent based on wrong decisions also vanish over time with the 
evaporation. In a traditional Linda-system, specialized processes have to 
take care of fault-detection and correction and often even the coordination 
language is extended. A SwarmLinda is fault-tolerant through adaptiveness, 
which also improves the availability of the system. Failures in nodes are 
treated by the ants as changes in the system’s configuration. 

Load Balancing: Adaptation to the current configuration of tuple production 
and consumption does not lead to bottlenecks in the system. As shown, the 
load in terms of activity of ants can also be dynamically adjusted by simple 
local decisions. 

4 Implementing SwarmLinda 

This section breaks down the issues one faces when given a task of implementing 
a SwarmLinda and discusses how the current prototype has been implemented 
and some of the alternatives that can be used to implement what we refer to as 
a full SwarmLinda. 

4.1 Network Topology 

In all the algorithms defined in Section 3, it is assumed that SwarmLinda nodes 
are organized in a terrain in which tuple- and template-ants can walk. However, 
a particular network topology is never mentioned. This was not by chance. The 
swarming only requires the nodes to be connected but their degree of connectivity 
(ie. how many other nodes they are directly connected to) is not specified. The 
connectivity of the nodes creates a neighborhood for each node. 

In the current implementation of SwarmLinda, the topology is fixed and the 
degree of connectivity of each node is uniform across all nodes. The nodes are 
organized in a two-dimensional grid as this reasonably represents a terrain used 
in ant-based algorithms. In practice though, the actual topology may not be as 
well organized with exactly N x M nodes since the system has to work with any 
number of nodes. 

A full SwarmLinda is not restricted to a N-dimensional grid nor to a uniform 
number of neighbors - nodes can keeps a dynamic list of neighbors. For a random 
walk, ants choose one of the entries in the list at random and use it as the 
direction for a next step. 

There is a tradeoff between the number of neighbors and the performance of 
the algorithms, since more neighbors lower the probability of taking the “right” 
direction by random. In a full SwarmLinda, this could be subject to another 
adaptation mechanism. The mentioned tradeoff can be quantified by introducing 
a notion of “fitness” of nodes. It is calculated from factors such as how often 
a node simply routes ants through to other nodes - slowing them down and 
lowering the fitness of the node - or how many matches occur at the node - 
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more matches raise the fitness factor. Based on such a factor, a node could for 
example tell its neighbors about shortcuts. 

This can be easily integrated to the dynamic update of routine entries. When 
a substantial number of ants coming from node A always go to node B but find no 
match there and go on to node C, node B can detect this and establish a shortcut 
from A to C. To do so, it tells the ants to change their origin information for 
the walk from B to C. If they carry A as the origin information, node C learns 
about A. When a first ant goes from C to A, node A will learn about C and 
a shortcut can develop. 

The maintenance of the neighbor list is essential for the algorithms and 
the method utilized must reflect the adaptiveness of the system and also take 
into consideration that nodes may be down or not working properly. Currently 
SwarmLinda implements something that is very similar to the algorithms used 
to keep routing tables in networks up-to-date. Each node maintains a list of 
nodes in its neighborhood. When a new node is started, it connects to another 
node which is already part of the current network and request its neighbor list. 
The new node then picks the node it connected to and N-l of its neighbors to 
build its own neighbor list. A neighbor list is also broadcasted to each of the 
neighbors when changes in the list need to be propagated. 

For a full SwarmLinda, the routing lists would also underlie the principles 
of dynamic adaptation. First of all, the neighborhood relation does not have 
to be implemented as symmetric. It can well be that node A knows of node B 
as a neighbor while B does not know about A. The movement of tuples itself 
would distribute the neighborhood information by carrying a reference to the 
node they came from. If an ant moves from A to B, it will tell B that it came 
from A. Then, B updates its neighborhood information accordingly. The benefit 
from this is the relaxation of consistency requirements among the neighborhood 
lists. Given a uniform tuple/ ant-movement, the lists will converge finally. 

The neighborhood information should not be fixed. An entry to A in the 
neighborhood list of B should vanish slowly over time unless it is refreshed by 
another ant/tuple coming from A. Also, at the system level, failed attempts to 
contact A should lead to an even stronger evaporation of the entry. Thereby the 
routing tables adapt to both failure and system communication structures. 

The details of the scheme and their calibration remain open. Critical is the 
initialization of the routing tables. One approach would be to have a new node 
N contact exactly one node M, being entered as a neighbor of M and copying its 
neighborhood list. N would be eventually entered into the neighborhood tables of 
the surrounding nodes. Another important issue is the control of lower and upper 
bounds in the number of neighbors of a node. Although it is desirable for the 
neighborhood configuration to be adaptable to changes, the general conditions 
of swarming may have to be relaxed to avoid undesirable extremes. 

4.2 Dealing with Separation of Concerns 

The algorithms described in Section 3 do not assume the existence of multiple 
tuple spaces in the system - at no point the algorithms refer to the logical loca- 
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tion of tuples. Does this mean that SwarmLinda is moving away from a known 
improvement to Linda systems? Not necessarily, although an implementation 
where multiple tuple spaces do not exist would be more straightforward, on the 
implementation level the use of multiple tuple spaces is not a complex task. 

The options available on how to implement multiple tuple spaces in the con- 
text of SwarmLinda are more limited than in standard implementation. Clearly, 
Linda process cannot (for certain) tell whether a tuple space is physically cre- 
ated. If the uniqueness of the handles is guaranteed one could easily implement 
a Linda system were all tuples are just prefixed with the tuple spaces handle 
instead placing them into a physical data structure representing the tuple space. 

Given that the swarm algorithms assume that all food is distributed in a flat 
structure, SwarmLinda would not be able to implement multiple tuple spaces 
by physically creating tuple spaces. The option then is to utilize the concept of 
prefixing all tuples with a handle - a process that is completely transparent to 
users. Therefore, in the implementation of SwarmLinda physical tuple spaces are 
not present. This is a consequence of the tuple distribution algorithm used in 
SwarmLinda which attempts to be adaptive and independent of any clustering 
imposed by the user such as the definition of tuple spaces. 

In a full SwarmLinda, however, the adaptation might take advantage of the 
fact that tuples in the same logical subspace “belong” together for some applica- 
tion specific reason. Therefore, it might be a useful optimization to use the handle 
when placing tuples or template. For example, the algorithm from section 3.2 
can be extended by weighting the handle-field very strong when determining the 
similarity of tuples. 

4.3 Movement of Ants 

All algorithms described earlier depend on the movement of tuple- and template- 
ants. The prime question here is how a decision can be made with regards to 
how ants decide their next step. In the currently prototype, the ants interact 
with the environment to decide what is the best suited neighbor node to move 
to. The suitability of the neighbor is computed based on two aspects: 

Scent strength of the neighbor : The scent strength for a particular tuple type 
(template) is kept as a characteristic of each node. When an ant decides 
to move to another node, it exchanges local messages with the neighboring 
nodes. The scent strength for that type is then used as a probability that 
the ant will choose to move to that location. 

Direction system: In some cases the scent alone may not have the desired effect. 
In fact, it is known that real ants also have an internal direction system. That 
is, even when scents are not available, ants tend to walk in straight lines - if 
an ant is moving towards direction D it is least likely to choose for the next 
step the direction —D (minus D). 

Since the environment of the current implementation is a two-dimensional grid, 
the direction system and the scent level can also be seen as a grid of values around 
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the current position of the ant (shown as shaded in Figures 5 and 6). Figure 5 
is an example of the direction system and scent levels currently implemented in 
SwarmLinda. 

The idea is simple, a direction system is pre-determined and fixed for the 
entire system. In swarming terms, this represents a system where all ants belong 
to the same species and therefore follow the same direction system. Currently, we 
are using a system of values that (under the absence of a scent) make an ant less 
likely to change its direction radically. For instance, observe the leftmost picture 
in Figure 5 and assume that the current direction of the ant is as indicated in by 
the arrow. Given these conditions, the direction system indicates that the ant 
has a 40/135 chance of keeping the same direction as opposed to 5/135 chance 
of choosing the exactly opposite direction. 

This system is always augmented with the information about the scent level 
of the neighborhood (middle picture in Figure 5). Once combined, they produce 
the final direction system which will be used for the next step of that particular 
ant. For instance in the scenario depicted, the ant has a 60/212 change of moving 
west and only 40/212 chance of keep moving in the same direction (north). 

Once the ant decides about the next move, the direction system may have to 
rotate to represent the initial idea correctly. In Figure 5 we argued that the reason 
for the higher chance of an ant moving north was due the fact that ants tend to 
keep the same direction. However, if it decides to take a different direction, the 
direction system must remain consistent with the change. For instance, assume 
that after the resulting values were calculated the ant indeed moved west. In this 
new position the direction system should reflect the fact that for the next step 
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the ant will have a 40/135 chance of keep moving west. The direction system 
rotates to the configuration shown in Figure 6. 

The relaxation of the approach to allow as many directions as necessary may 
hinder the use of the direction system idea as described above. Following the 
“swarming” idea, one could adapt the direction system to a system of success. 
Note that the scent is not an indication of the path taken by the previous ants 
but rather the path used by returning ants. So if we assume no evaporation 
of scents, a scent of value S indicates that S ants returned via that node after 
successfully finding the tuple/template they looked for. This does not necessarily 
mean that S ants took that path before finding the tuple/template. 

Indeed the scent has an influence on the decision of an ant but we also want 
to have a secondary value that indicates the probability of an ant to move in 
situations where scent values are in existent or too similar. A secondary scent-like 
value can be stored in the environment keeping track on the number of ants that 
took that particular direction. This should be represented using a delta vector 
(see description in the next section) . For instance, a vector could have the values 
(12,5,1,3,4) indicating that 12 ants that reached that node decided to go to 
neighbor 1, 5 ants decided to go to neighbor 2, and so on. These values can 
now be used in a similar fashion as the direction system described above - used 
as probabilities. In this example ant ant would have 12/25 chance of going to 
neighbor 1, 5/25 chance of going to neighbor 2, and so on. 

4.4 Ant Activity and the Environment 

The environment keeps a state which can be sensed by the ants. It has several 
dimensions, making it possible to drop different kinds of scents for different 
information. The state observable at a place in the neighborhood of an ant is 
a vector consisting of intensity values of different scents. We currently assume 
a fixed dimension of this vector. 

Every ant operates in a cycle: (i) observe environment, (ii) plan change of 
environment and (in) change environment. The observation results in a vector 
for each neighborhood environment location. A complete observation leads to 
a matrix of vectors which represents the environment-matrix at the time of 
observation. 

The ant then has to generate a delta- vector that reflects the changes planned 
to the environment. To drop a unit of scent of kind 2 at the current location, the 
respective delta vector could be (0, 1, 0, 0) for a four dimensional state. All delta- 
vectors together form a delta-matrix. In the final step, the change is applied by 
adding the delta-matrix to the environment matrix in an atomic step. 

While in the current algorithms described ants always add one unit of scent 
to their environment, we do not restrict our change system to that. Ant that 
follow other algorithms could also drop five units of scent with a vector (0, 5, 0, 0) 
for example when they perform a collect operation and carry more than one 
tuple. 
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The atomic addition of the delta matrix ensures that ants can operate concur- 
rently. In some future algorithms they can even neutralize changes, for example 
when another ant adds the delta vector (0,— 5,0,0) to the current location. 

The evaporation of scent is implemented by periodic application of vectors 
on each location, for example (— 1, — 1, — 1, — 1) if all scents are greater that 0. 
Spreading scent can be implemented by a similar delta-matrix. 

4.5 Lifecycle of Ants 

The description above makes it clear how ants communicate with the environ- 
ment but one still needs to discuss when they are created and why do they cease 
to exist (die). 

One can consider ants as mobile agents (or mobile processes) . Although this 
is an implementation choice, it is not necessarily the only one. In fact, in the 
current implementation ants are messages transmitted between the nodes of the 
network. 

Tuple-ants are messages created when “storing” primitives are executed (eg. 
out). Once a tuple-ant is created, it is inserted in the network from some initial 
node. From that point onwards, the node will decide whether the ant should drop 
its food (the tuple contained in the message) locally or whether the message 
should be transmitted to one of the neighbor nodes. The decision follows the 
algorithms described in Section 3. 

Each tuple-ant has a delta value that is incremented every time the ant 
is routed to a new node. The larger this delta value becomes the greater is 
the likelihood the tuple will be stored at that location. Template-ants on the 
other hand, are created when a “retrieving” primitive is executed (eg. in, rdp). 
Similarly to tuple-ants, these are also implemented as messages that are routed 
through the nodes. The difference here is the consequence of the delta value. 
Unlike the case with tuple-ants, template-ants cannot just be abandoned (or 
stored in the case of tuples). In Linda-systems, a request for a tuple must be 
fulfilled (or fail in the case of non-blocking primitives) . 

In the current implementation, the delta-value is used to determine failure. 
If a template-ant dies, it reports the failure to the originating process which uses 
the failure message as the return value for the primitive executed (in the case of 
inp, rdp). 

For the case of primitives that must be fulfilled the template-ant behaves as in 
the algorithm described in Section 3.1. If a template-ant reaches a failure mode, 
the node places the message in a “sleep list” . This represents the templates-ants 
that could not find a matching tuple. The sleep time may allow the status of 
the system to change and matching tuples to be stored. After a few unsuccessful 
sleeps, the template-ant is destroyed and the message is recreated in the original 
source. Another option after several recreations is for the template-ant to “warp” 
to a completely random node somewhere and restart the search algorithm there. 
This last step has not yet been implemented in SwarmLinda as its benefits are 
not yet clear since it can create “false” paths. 
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In future implementations, we want to deal better with Linda specific char- 
acteristics such as blocking. Currently blocking is “simulated” but not really 
implemented. As described in above ants will keep searching until a matching 
tuple is found. A problem that might happen with this approach is that if a tuple 
does not really exist in the system, the approach where templates keep looking 
for it may be naive and costly. A possible approach is to use the aging mechanism 
embedded in template-ants to decide that the template should, instead of keep 
searching, stop and diffuse its scent at that location (characterizing the blocking) 
thus not wasting the system’s resources. Future tuple-ants should then be able 
to track the scent of the template-ants and unblock them if a match takes place. 

5 Conclusion 

This paper described approaches for implementing a Linda systems using swarm 
intelligence, SwarmLinda. The paper demonstrated that organized behavior in 
SwarmLinda can be implemented based on a few simple rules that mimic natu- 
rally forming multi-agents systems. We claimed that the use of swarm abstrac- 
tions in an implementation of SwarmLinda would not only be simple but will 
improve its scalability and adaptiveness. 

The algorithms presented here provide an alternative to the standard ap- 
proaches used in various Linda implementations. The algorithms follow three 
principles that can be observed in most swarm-based systems: simplicity, dy- 
namism and locality. 

Implementations and other publications related to SwarmLinda will be made 
available in the SwarmLinda Project website [18]. 
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Abstract. How should open agent societies be organized? Should they 
be democracies, and, if so, what types of democracy? We present three 
normative models of democracy from political philosophy and consider 
their relevance for the engineering of open multi-agent systems: democ- 
racy as wise rule by an elite; democracy as the exercise of rational con- 
sumer choices by voters; and democracy as deliberative decision-making 
by citizens. We consider the implications of these different models for the 
design of open systems, in terms of the communications language, the 
interaction protocol, and the conflict-resolution mechanism used by the 
agents involved. We also consider the issue of verifiability of the internal 
semantics of communications languages, and argue that a model of agent 
democracy based on deliberative democracy provides the basis for a form 
of verifiability which is stronger than a social semantics. 



1 Introduction 

Open agent systems are multi-agent systems with open admissions policies and 
therefore potentially fluid membership. Entry may require compliance with par- 
ticular stated conventions, such as use of an agent communication language and 
interaction protocol, or the making of a financial deposit. However, subject only 
to such conventions, any agent may join. Because such agents may represent 
different human principals and typically will have been constructed by different 
software design teams, they may, in general, have conflicting goals, interests, 
beliefs and values. In these circumstances, which agent’s goals or beliefs prevail 
in the interaction will depend on the nature of the social and political relation- 
ships between the participants. In situations where the agents adhere to some 
hierarchical relationship inside the agent system, that agent or agents at the top 
of the hierarchy may have final decision-making authority. For example, in an 
auction interaction, the auctioneer may have the explicit power to determine the 
final allocation of the scarce resources being sought by the bidders. Power such 
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as this may not reside in particular agents, but accrue to certain roles within 
the agent system, as in the electronic institutions of [1]. 

However, if human interaction is any guide, in many open environments there 
will either be no such hierarchy, or what hierarchies there are may be contested 
by some participants. Indeed, this is already true of existing agent societies on 
the Internet. What structures are appropriate for agent societies in these cir- 
cumstances? The absence of hierarchy means that the relationship between the 
participants is closer to one of equality; this in turn suggests that some form of 
democracy is appropriate when we consider the structure of these agent systems. 
Within the discipline of political philosophy, human democracy is a notion much 
debated, and there are several alternative normative theories of democracy [2]. 
In this paper, we explore these alternatives from political theory, in order to 
identify what structures they provide for, and what constraints they place on, 
designers of multi-agent systems. A designer of an open agent system intending 
to permit democratic participation by the agents in the system therefore has 
a choice of forms of democracy to encode in the system; indeed, an agent society 
may encode forms other than those studied by political philosophers. In Sec- 
tion 2 we present the three primary normative theories of democracy developed 
by political philosophers, and then discuss, in Section 3, their implications for 
the design of open agent systems. It happens that one theory, the Deliberative 
Model of democracy, stresses the joint and discursive nature of decision-making 
in a democracy, with participants exchanging arguments for and against vari- 
ous policy proposals, and forming preferences on the basis of these exchanges. 
The structure that this model provides to the agent system designer creates the 
means necessary to develop a strong form of social semantics, thereby increasing 
the extent to which a mentalistic semantics of an agent communications lan- 
guages can be verified. This view is explained in Section 4. The paper concludes 
with a summary in Section 5. 

2 Three Models of Democracy 

Political philosophers have articulated several normative models of democracy, 
and we present here the three most influential. The problem they confront was 
first formulated in an abstract form by philosopher Jean- Jacques Rousseau [3], 
who viewed a polity as comprising just two entities: Society and the State. Society 
is the collection of individuals, organizations and companies in a polity, together 
with the panoply of relationships between them, while the State is the apparatus 
of public-sector administration. The key question for political theory is then: 
What should be the process of formation of political will? or How should Society 
program the State? 1 Supporters of democracy believe that these questions should 
be answered with the use of democratic procedures, such as elections based on 
universal adult suffrage. But if such procedures are used, what is the nature 
of the relationship between citizens and their elected representatives? Rousseau 

1 Note that use of the word “program” in this context is not due to our computational 
perspective, but is the usage of political philosophers [ I, p. 239]. 



68 



Peter McBurney and Simon Parsons 



had assumed that the people have a single “general will” which their elected 
representatives should seek to implement, but this is at best only a high-level 
approximation to the multifarous cacophony which is modern democracy. 2 

The first modern political theory of democracy which sought to answer this 
question was proposed in 1942 by Austrian- American economist Joseph Schum- 
peter [5]. Schumpeter’s theory, possibly in reaction to the mass populism of 
Nazism and Communism and to his own failed political career, was disdainful of 
ordinary people and their views: “Thus the typical citizen drops down to a lower 
level of mental performance as soon as he enters the political field. He argues and 
analyzes in a way which he would readily recognize as infantile within the sphere 
of his real interests. He becomes a primitive again/’ [5, p. 262]. Consequently, 
Schumpeter proposed that elected officials should act as a technocratic elite, 
making decisions on behalf of the general public and in accordance with what 
the elite believes are the public’s best interests. Apart from voting, the people are 
entirely passive in Schumpeter’s model of democracy, which has rightly acquired 
the label elitist [2, p. x]. We call this the Wise Elite Model of democracy. 

In contrast to Schumpeter’s hierarchical view of democracy, Anthony Downs 
proposed an economic-theoretic model of political will-formation in a democ- 
racy in which citizens were more than simply passive objects [6]. This model 
has since been called a rational- choice or liberal model [4], and it views democ- 
racy as akin to the operation of an economic market. Downs proposes a theory 
of democracy where political parties and interest groups act as entrepreneurs, 
offering alternative “products” in the form of bundles of state-instructions (or 
equivalently, ideologies, which are philosophies of bundle- formation) , to voters 
who then “purchase” their preferred bundle when they vote. That bundle with 
the greatest “market-share” — in the form of popular votes — becomes the 
set of instructions used to program the State. Downs explicitly assumed that 
voter-consumers in a free and democratic society make their political choices on 
the basis of their perceived self-interest, and act according to the now-standard 
definition of rational economic behavior, e.g., [7, 8]. In other words, voters are as- 
sumed to always vote so as to maximize their perceived expected utility from the 
outcome of the election. In addition to consuming bundles of state-instructions, 
citizens also consume information about policies, ideologies, political parties and 
candidates to the extent necessary to make their voting decisions. And, as for any 
other good, such consumption may be subject to time-, resource-, or processing- 
constraints, and cost-benefit trade-offs. 

The rational-choice model affords citizens a greater role than does the wise 
elite model, namely that of consumers of relevant political information and of 
recipients of the effects of policies enacted by their representatives. But citizens, 
in the rational-choice model of democracy, are not regarded as producers of polit- 
ical information or public policies. This viewpoint ultimately stems, we believe, 
from Downs’ adoption of Kenneth Arrow’s operational definition of economic 

2 Rousseau gave no procedures for identifying the general will, nor for reconciling 
competing interpretations of it. Accordingly, we do not discuss his model further 
here. 
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rationality [7], which assumes that a decision-maker’s preferences and utilities 
are given and precede the task of selection of a decision-option. In many, if not 
all, public policy determinations, however, the preferences and utilities of vot- 
ers may only be formed in the very process of decision-making, as participants 
learn about feasible decision-options and about the effects of various decision- 
options on one another and on others not involved in the decision process, as 
argued in [9] . Moreover, to the extent that one person’s utility depends on 
the welfare or preferences of others, a rational, resource-unconstrained, voter 
would not finally determine his or her preferences until hearing from those oth- 
ers about their own utilities and preferences. 3 Insofar as preferences can only be 
determined jointly, or require information not available to everyone, multi-party 
deliberation therefore cannot be undertaken by individuals reasoning alone, as 
the rational-choice model assumes. 

In contrast, the deliberative democracy model of political will-formation em- 
phasizes the manner in which beliefs and preferences of participants are formed 
or change through the very process of interacting together [11, 12]. In this model, 
citizens do not merely interact to exchange their preferences at election time, and 
to consume political information, as is the case with the rational-choice model. 
Rather, they are also producers of political information and policies, as they par- 
ticipate in political processes and debate, identify and publicize issues of personal 
or social concern, exchange arguments for and against various policy options, and 
generally seek to influence the outcomes of political decision processes. Seeking 
to influence and persuade other participants means that they must themselves be 
open to persuasion, and thus undergo what has been called self- transformation 
[13, p. 184]. Although few studies have been conducted, there is evidence that 
deliberative decision-processes lead to better decision outcomes [14, 15]. 

The rational-choice and deliberative models of democracy embody different 
notions of rationality. As mentioned above, Downs’ economic theory of democ- 
racy was based explicitly on Arrow’s [7, Chapters 1 and 2] definition of rational 
behaviour as the maximization of expected perceived utility. This in turn was 
an operationalization of Lange’s notion of rationality [8, p. 30]: “A unit of eco- 
nomic decision is said to act rationally when its objective is the maximization of 
a magnitude. ” This notion of rationality, although predominant in economics, 4 
is not how the word is understood in the philosophy of argumentation [19, 20]. 
For example, Ralph Johnson [20, p. 14] gives this definition: “Rationality is the 
ability to engage in the practice of giving and receiving reasons. ” The deliberative 
model of democracy, because it construes democracy as the joint determination 
of public policy by the citizenry through debate and exchanges of views, embod- 

3 For example, one person’s utility from a so-called network good, such as a fax ma- 
chine, depends on whether or not other people have them. In so far as a purchase 
by one consumer sends signals to others [10], all products may be viewed as network 
goods to some extent. 

4 And influential in AI. It has been subject to much criticism, however; e.g., [16, 17]. 
Indeed, a winner of the Nobel Memorial Prize in Economics, Amartya Sen, has 
recently argued for a notion of rationality in economic theory which is closer to that 
used in the philosophy of argumentation [ 8]. 
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ies the philosophers’ notion of rationality rather than the economists’, although 
the latter is not precluded. For agents who know the reasons for their own ac- 
tions, the maximum-expected-utility notion of rationality is a special case of 
argumentation-theoretic rationality, since the method used to select an action- 
options can be advanced as a reason for the option by a speaker in a debate. 
However, the converse is not true: there may be many reasons for selecting an 
option which does not maximize expected utility, e.g., that it avoids catastrophic 
downside loss. 5 

These three models of political will-formation in a democracy can be seen as 
offering alternative roles to the citizens who comprise the Society. In the Wise 
Elite model, the people are seen as completely passive, except when choosing 
the Elite. In the Rational-Choice model, the people are viewed as consumers 
of policies, ideologies and information. In the Deliberative model, the people 
are viewed as both consumers and producers of policies, ideologies and political 
information. 

3 Design Implications for MAS 

Which of these theories of democracy is appropriate for the design of open multi- 
agent systems? The answer, of course, depends on the intended objectives of 
the system, and the nature of the application. In systems with differentiated 
agent roles and an explicit hierarchy some version of the Wise Elite model of 
democracy may be appropriate. An example here are public auction sites, in 
so far as the auctioneer makes decisions on behalf of the agents comprising the 
auction. Bidders have the freedom to join or not any given auction site, and so 
may be viewed as “electing” the elite in the form of the auctioneer. Once the 
auction is underway, however, bidders express preferences through their bids, and 
the rules of the auction mechanism may resolve these into a collective decision, 
rather than leaving the resolution to the wisdom of the auctioneer. Another 
example of the Wise Elite model is the use of agents for security and access- 
policing functions, similar to the space-administration objects of [21]. One can 
readily imagine the participants in an open agent system agreeing to delegate 
a limited amount of their joint power to a group of policing agents, who exercise 
that power in pursuit of collective aims of security and confidentiality which all 
agree are essential. For example, the interaction protocol rules may permit any 
participant to speak at any time, but a policing agent could prevent participants 
monopolizing the microphone by limiting usage from verbose (or badly-coded) 
agents. Thus, the collective goal of fair distribution of microphone access takes 
precedence over an individual agent’s goal of exclusive use of the microphone, 
and the policing agents act to ensure this on behalf of all agents. 

However, in most open agent systems, such voluntary ceding of power will 
not occur on matters of concern to the participants. Participants are likely to 
disagree with one another over such issues, and will wish to express their own 

5 Habermas [19, p. 10] calls these two notions of rationality cognitive-instrumental and 
communicative , respectively. 
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beliefs, preferences and intentions. Agent autonomy means that software agents 
cannot in general be ordered to fulfill requests; they may be requested and, at 
best, persuaded to do so. If their beliefs, preferences and intentions are pre- 
determined and fixed, no amount of persuasion will change these, and so the 
Rational Choice model of democracy would be appropriate. Here the only sen- 
sible interaction mechanism between the participating agents would be some 
form of preference aggregation or voting, since the exchange of reasons for be- 
liefs or preferences would not alter decision outcomes. Auction mechanisms are 
examples of open multi-agent societies where agents are usually assumed to have 
pre-determined (although not necessarily fixed) preferences, and where no party 
seeks to persuade another to change these. In this case, democracy as the ex- 
pression of preferences, as in the Rational Choice model, is sufficient to represent 
the relationships between the agents concerned. 

In other domains, however, agents may well seek to influence the beliefs, 
preferences or intentions of others. Whenever the relationship between agents 
in a multi-agent system is one of equality, and where agents seek to influence 
the beliefs, preferences or intentions of one another, then a Deliberative model 
of democracy will be the most appropriate model for the design of the sys- 
tem. Adoption of a particular model of democracy for a multi-agent system has 
a number of design implications, which we explore in the next three subsections. 

3.1 Communications Languages 

The different models of democracy place different requirements on the commu- 
nications language required for agent interaction. The Wise Elite Model requires 
that agents have some means to select the elite. But, other than this, no other 
expressions of beliefs, preference, intentions, etc, need be expressed by the par- 
ticipants, since all decisions are made by the elite. Under the Rational Choice 
model, participants express preferences for or between policy options but not 
necessarily arguments for these preferences. Thus, the communications language 
needs to be able to support the expression of preferences, either directly or by 
means of acceptance or rejection of particular proposals. Auction protocols, such 
as the FIPA ACL English auction protocol [22], typically permit the expression 
of preferences through utterances of acceptance of particular proposals. 

Adoption of a deliberative model for democracy in a multi-agent system re- 
quires that the communications language allows each agent to express, not only 
its beliefs, intentions or preferences, but also its arguments for or against these. 
Participants require the ability to question or challenge the statements of oth- 
ers, and to defend and justify their own statements. Thus, the communications 
language needs to be able support the expression of arguments for statements, 
as well as expression of the statements themselves. There have been a number of 
proposals for agent communications languages providing this capability in recent 
years, e.g., [23, 24, 25]. 

In addition, if participants are to engage in debate with the potential to 
persuade one another to adopt new beliefs, intentions or preferences, then agents 
need to be able to withdraw prior statements and utter replacements in their 
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stead. If one agent’s goal in an interaction is to persuade a second agent of some 
belief which that other currently does not endorse, then the communications 
language should enable the second to express any changes of belief it makes as 
a result of the interaction; otherwise, why would the first agent seek to persuade 
the second? This self-transformation capability has not typically been a feature 
of agent communications languages or interaction protocols. In recent work [26], 
we proposed expression of self-transformation as a design criterion for multi- 
agent systems using dialogue game protocols, and assessed various interaction 
protocols and languages against this criterion (among others). As noted there, 
the FIPA Agent Communications Language FIPA ACL [27] provides only limited 
capability for participants to question one another and to express any changes in 
beliefs. Because FIPA ACL lacks retraction illocutions, changes in agent opinions 
can only be expressed by successive, and possibly contradictory, utterances of 
belief. An agent who believes the sky is blue utters an inform illocution to this 
effect; if it subsequently comes to believe the sky is red, it can only express 
this with a second, contradictory, inform illocution. How is a listener to such 
a sequence of contradictory utterances to interpret it? The sequence may be 
evidence of updated beliefs by the speaker, or it may be due to malice, whimsy, 
or simply faulty code. Explicit retraction locutions can ensure no such ambiguity 
of interpretation. Agent languages based on formal dialogue games have a better 
record in this regard [26], perhaps due to their origin as protocols for the conduct 
of debates in philosophy. However, there is still much work to be done in this 
area, particularly for dialogues over action rather than belief [28, 29]. 6 

3.2 Interaction Protocols 

We distinguish between the communications language used by the agents in an 
agent society to make utterances, and any rules which govern the combination 
of utterances, which, when combined with the language, we call the interaction 
protocol [31]. The FIPA ACL [27], for example, has no such rules, with the result 
that any utterance by any agent may follow any other by any agent. Of course, 
such rules may defined to overlay the FIPA ACL, as with the various auction 
protocols defined by FIPA, e.g., [22]. As with the communications language, the 
requirements placed on any interaction protocol will differ according to which 
model of democracy is used. Because the Wise Elite model does not require any 
expression of opinions or of arguments, there are no requirements on the inter- 
action protocol. The Rational Choice model only requires expression of opinions 
or preferences, and so any interaction protocol would need to enable expression 
of these in an orderly fashion. Auction protocols, for example, typically proceed 
through a series of rounds, with constraints on what can be uttered at each 
round [22]. 

The Deliberative model, by contrast, leads to the most extensive requirements 
on any interaction protocol. If participants are able to question and challenge 

6 We note in passing that threats and rewards may be used effectively to change 
a person’s proposed actions, but not their sincere beliefs, a situation called Pascal’s 
Law by Cristiano Castelfranchi [30]. 
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one another’s statements, and to defend their own statements when challenged, 
then an orderly interaction will require rules relating one type of utterance to 
another, and specifying when particular utterances are required or prohibited. 
For instance, the rules may specify the circumstances under which a question 
seeking the reasons for some claim must be answered by the agent which made 
the claim; without such a rule, the questioner would have no guarantee that the 
question would receive a response. Interaction rules such as these have received 
considerable attention from philosophers of argumentation, e.g., [32, 33, 34], 
work which has, in turn, influenced the design of agent interaction protocols, e.g., 
[23, 35]. Recent work, for example, has considered the formal specification [31] 
and verification [36] of agent interaction protocols. 



3.3 Resolution Mechanisms 

Agents in a multi-agent system may have different beliefs and intentions; ac- 
cordingly, agent researchers have designed mechanisms to enable agents to share 
their opinions and justifications, and to engage in persuasion and negotiation 
dialogues, e.g., [25]. However, differences of opinion may persist even after ex- 
changes of justifications and attempts at persuasion. In circumstances where 
a collective judgment must be made, such as where a group of agents need to 
agree a joint course of action, then the agents require some mechanism for resolv- 
ing their differences. The mechanisms which are feasible and appropriate differ 
according to the model of democracy used. 

Under the Wise Elite model, all decisions are taken by the elite so, as far 
as the other participants concerned, there is no need for a conflict-resolution 
mechanism. 7 In an auction with one seller and many potential buyers, for exam- 
ple, the auctioneer determines the winner, usually (but not necessarily) on the 
basis of previously-published rules; the auctioneer thus resolves the difference 
of opinion between the buyers unilaterally. Under the Rational Choice Model, 
agents choose between policies as if the agents were consumers and the policies 
were products. Although agents may receive information about policy options, 
the Rational Choice model does not assume they necessarily engage in debate 
or argument about these. The final resolution of any differences is made by each 
agent choosing whichever policy it most favors. If these choices differ, then the 
appropriate resolution mechanism is a vote by the agents, selecting that pol- 
icy, for example, with the greatest numerical support. For instance, Hunsberger 
and Zancanaro propose a voting procedure for pooling agent judgements over 
alternative partial plans in undertaking joint planning activities [37]. 

Under the Deliberative Model, however, it is assumed that agents may engage 
in debate over policy choices, and so there may be an exchange of arguments prior 
to determination of a collective judgment. As with the Rational Choice Model, 
a voting mechanism may be used to make this final determination. But the ex- 
change of arguments means that other mechanisms are also feasible, relying on 
the argument-aggregation procedures from argumentation theory, e.g., [38, 39]. 

7 The Elite itself, if comprised of more than one agent, may require such a mechanism. 
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An example of such a procedure may make this clear. In [39], claims are classi- 
fied into one of several mutually-exclusive classes on the basis of the arguments 
presented for or against them. In this framework, one argument B rebuts an- 
other A if B is an argument for the claim -i 9 and A is an argument for the claim 
9. An argument C undercuts A if C is an argument for a claim -17, where 7 is 
a premise of argument A. We can then define the argument-status of a claim 0 
at time t as follows: 

— If there have been no arguments uttered for or against 9 up to time t , then 
the claim is Open. 

— If there has been at least one argument uttered for 9 up to time t, then the 
claim is Supported. 

— If there has been at least one argument with consistent premises uttered for 
9 up to time t , then the claim is Plausible. 

— If there has been at least one argument whose premises are consistent uttered 
for 9 up to time t, and no undercutting or rebutting arguments have been 
uttered against 9 by this time, then the claim is Probable. 

— If there has been at least one argument whose premises are consistent uttered 
for 9 up to time t, and any undercutting or rebutting arguments uttered 
against 9 by this time have themselves been rebutted or undercut, then the 
claim is Confirmed. 

The motivation for this approach is that the more and the stronger are the 
arguments for a claim, then the more support it has among the participants. 
The labels used, Open, Supported , etc, are entirely arbitrary and any set of 
qualitative labels could be defined in this way. Such a classification of arguments 
can be used as a conflict-resolution mechanism when the agents concerned are 
unable to agree on a claim. In [35], we explored the formal properties of such 
an argument-based resolution mechanism, particularly the circumstances under 
which the labels assigned to a claim would converge over time. 8 

Some political theorists claim that being open to persuasion requires par- 
ticipants with conflicting views to see each other as adversaries rather than as 
enemies, engaged in argument in particular interactions in the joint knowledge 
that every interaction may be followed by others [40, 41]. Participants therefore 
need to achieve a feasible middle ground between striving for an impossible con- 
sensus and refusing to interact with one another. Accordingly (these theorists 
argue) , democratic political institutions and processes need to be able to permit 
participants to express what may be very different preferences and goals, and 
to participate in joint political processes despite such differences. Argument- 
classification systems, such as the one above, facilitate this by incorporating all 
the views and arguments expressed, even those which are not in the majority. 

In summary, the model of democracy adopted has implications for the types 
of mechanisms which are feasible for resolving conflicts of agent opinion. Such 



There is still work to be done, however, on the rate of convergence, so as to better 
understand when in a dialogue it is appropriate to deploy a resolution mechanism. 
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differences of opinion are ignored under a Wise Elite model. Voting is essen- 
tially the only mechanism possible under a Rational Choice model, while the 
exchange of arguments under a Deliberative Model enables the additional use in 
these systems of resolution mechanisms based on argument classification from 
argumentation theory. 

4 Semantic Verifiability 

Verification of semantic requirements of agent communications languages 
and interaction protocols is problematic [42, 43]. This is essentially because 
a sufficiently-clever agent can always simulate insincerely any required men- 
tal state. In response to this problem, Munindar Singh [44] proposed a social 
semantics for agent communications languages, in which each participant to an 
interaction makes a public statement of its beliefs and intentions. Other agents 
can then use these public declarations to ensure that each agent is consistent in 
its utterances in an interaction. Of course, an agent may still make insincere dec- 
larations, but at least it can be called to account for inconsistencies between its 
declarations and its subsequent statements, as one of us has formalized in [45]. 
In the vernacular: Liars need good memories. 

If a deliberative model of democracy forms the basis of an open agent sys- 
tem with a social semantics, then we are able to obtain a stronger form of 
semantic verifiability. Under a deliberative model, agents making claims may be 
questioned and challenged by other agents about the reasons for those claims; 
these reasons could be arguments for a belief of the agent, or reasons for an 
intention. Consequently, not only can the consistency of declarations and other 
utterances in the interaction be verified, but also the degree to which the decla- 
rations themselves - beliefs or intentions — are justified. We call this form of 
verification contestability , since social semantic declarations may be contested or 
challenged by other agents. 9 Of course, insincere declarations are still possible, 
but agents making false declarations may also need to fabricate a set of argu- 
ments for them. To be convincing to others, an insincere agent needs to create 
a set of inter-locking arguments for its statements, and other agents may only 
accept these arguments in defined circumstances, as is the case with the formal 
argumentation systems of [45, 46]. For example, if the listeners are skeptical 
regarding what arguments they accept [46], then they will only accept those 
arguments which defeat (in a precise sense) any attacking argument. Creating 
a set of interlocking arguments which convince a skeptical agent will usually not 
be easy for an insincere agent. In the words of Walter Scott [47, Canto 6, Stanza 
17]: “Oh what a tangled web we weave , when first we practice to deceive /” 

9 We also use this word because of the analogy with its meaning in economics. A con- 
testable market is one to which new entrants may join at any time, a prospect which 
should lead existing self-interested suppliers to act as if competitors were already 
present. Thus a monopolist in a contestable market may behave as if in a competitive 
market. 



